Display system, electronic device, and display method

ABSTRACT

A display system including a PC and an HMD, and in the display system, the PC includes an I/F unit configured to output image data, the HMD includes an I/F unit configured to acquire the image data output by the PC, an image display unit configured to superimpose an image on an outside scene visually recognized in a state where the HMD is worn, and to display the image, and a display controller configured to cause the image display unit to display the image data acquired by the I/F unit, and the display controller causes the image display unit to display the image data in correspondence to a position of the PC visually recognized as an outside scene.

This is a Continuation of U.S. application Ser. No. 16/228,847 filedDec. 21, 2018, which in turn claims the benefit of: (a) JP 2018-172773filed Sep. 14, 2018; and (b) JP 2017-246853 filed Dec. 22, 2017. Thedisclosure of the prior applications is hereby incorporated by referenceherein in its entirety.

BACKGROUND 1. Technical Field

The invention relates to a display system, an electronic device, and adisplay method.

2. Related Art

In the related art, a display device configured to receive and display adisplay image of an external device has been known (for example, seeJP-A-2015-227919). JP-A-2015-227919 describes an example in which, tointegrate display screens of a plurality of devices, a head mounted typedisplay device (Head Mounted Display: HMD) receives a display imagetransmitted by an external device and causes a display unit to displaythe display image.

In a configuration disclosed in JP-A-2015-227919, assuming that all aredisplayed by the HMD, the display image of the external device can beintegrated in display of the HMD. In contrast, in the related art, thereis no example proposed for a technique for controlling display by adisplay device in correspondence to an external device different fromthe display device.

SUMMARY

An advantage of some aspects of the invention is to control display by adisplay device in correspondence to an external device.

To address the above-described issue, a display system according to anaspect of the invention includes an electronic device including a firstdisplay unit, and a display device having a head mounted type andconnected to the electronic device, and in the display system, theelectronic device includes an output unit configured to output an image,the display device includes an acquisition unit configured to acquirethe image output by the electronic device, a second display unitconfigured to superimpose an image on an outside scene visuallyrecognized in a state where the display device is worn, and to displaythe image, and a display controller configured to cause the seconddisplay unit to display the image acquired by the acquisition unit, andthe display controller is configured to cause the second display unit todisplay the image in correspondence to a position of the electronicdevice visually recognized as an outside scene.

According to the aspect of the invention, when the image output by theelectronic device is displayed by the display device having a headmounted type, the image is displayed to correspond to the position ofthe electronic device as the outside scene. For example, the image canbe displayed by the display device in correspondence to a position ofthe first display unit visible as the outside scene. Accordingly,display in correspondence to the position of the electronic devicevisually recognized as the outside scene can be performed by the displaydevice configured to superimpose the image on the outside scene and todisplay the image.

Additionally, according to an aspect of the invention, the electronicdevice includes an electronic device control unit configured to generatean image corresponding to a virtual display area wider than the firstdisplay unit, cause the first display unit to display a portion of theimage generated, and cause the output unit to output at least a portionof the image generated, and the display controller provided in thedisplay device is configured to cause the second display unit to displayat least a portion of an image output by the electronic device, incorrespondence to a position of the electronic device.

According to this configuration, the second display unit of the displaydevice can be caused to perform display corresponding to the virtualdisplay area wider than the first display unit of the electronic device.Thus, a display area larger than the first display unit can be displayedin correspondence to the position of the electronic device, and thefirst display unit can be expanded virtually by the display device.

Additionally, according to an aspect of the invention, the electronicdevice control unit is configured to cause the output unit to output theimage obtained by removing a portion displayed by the first display unitfrom the image generated in correspondence to the virtual display area,and the display controller provided in the display device is configuredto cause the second display unit to display an image output by theelectronic device.

According to this configuration, when the display device performsdisplay corresponding to the virtual display area wider than the firstdisplay unit of the electronic device, a portion of the image isdisplayed by the first display unit, and a portion obtained by removingthe portion displayed by the first display unit is displayed by thesecond display unit. Thus, display including display of the electronicdevice combined with display of the display device can be achieved. Forexample, a display mode in which an image is displayed around theelectronic device by the second display unit and the first display unitis virtually expanded can be achieved by the display device.

Additionally, according to an aspect of the invention, the displaycontroller provided in the display device is configured to cause theimage output by the electronic device to be displayed around the firstdisplay unit visually recognized as an outside scene.

According to this configuration, the image is displayed by the seconddisplay unit of the display device around the first display unit of theelectronic device, and accordingly, the display mode in which the firstdisplay unit is virtually expanded can be achieved.

Additionally, according to an aspect of the invention, the electronicdevice control unit is configured to, based on a relative position ofthe first display unit with respect to the second display unit, causethe first display unit to display a portion of the image generated incorrespondence to the virtual display area, and cause the output unit tooutput the image obtained by removing a portion displayed by the firstdisplay unit.

According to this configuration, the electronic device outputs the imagedisplayed by the display device to correspond to the relative positionof the first display unit with respect to the second display unit. Thus,an operation in which display corresponding to the position of the firstdisplay unit of the electronic device is performed by the display devicecan be achieved easily.

Additionally, according to an aspect of the invention, the electronicdevice control unit is configured to, based on a relative position ofthe first display unit with respect to the second display unit, causethe output unit to output an image obtained by masking a portiondisplayed by the first display unit, of the image generated incorrespondence to the virtual display area.

According to this configuration, since the portion displayed by thefirst display unit of the electronic device is not displayed by thesecond display unit, the image displayed by the electronic device andthe image displayed by the display device can be coordinated.

Additionally, according to an aspect of the invention, the electronicdevice control unit is configured to cause the output unit to output theimage generated in correspondence to the virtual display area, and thedisplay controller provided in the display device is configured to causethe second display unit to display an image obtained by cutting out aportion of the image output by the electronic device.

According to this configuration, in processing by the display device, aportion of the image displayed by the electronic device can be expandedand displayed by the second display unit.

Additionally, according to an aspect of the invention, the displaycontroller is configured to, based on a relative position of the firstdisplay unit with respect to the second display unit, extract a portionof the image acquired by the acquisition unit, and cause the seconddisplay unit to display the portion.

According to this configuration, since the display device determines therelative position of the first display unit with respect to the displayunit, and generates an image for display to correspond to this relativeposition, display corresponding to the position of the first displayunit can be achieved without increasing a load on the electronic device.

Additionally, according to an aspect of the invention, the displaycontroller is configured to, based on a relative position of the firstdisplay unit with respect to the second display unit, cause the seconddisplay unit to display an image obtained by masking a portionsuperimposed on the first display unit, of the image acquired by theacquisition unit.

According to this configuration, in processing by the display device,since the portion displayed by the first display unit is masked in theimage displayed by the second display unit, the image displayed by theelectronic device and the image displayed by the display device can becoordinated without increasing a load on the electronic device.

Additionally, according to an aspect of the invention, the electronicdevice control unit is configured to determine a position of the virtualdisplay area, based on a position of the electronic device in a realspace, and adjust a display mode of the second display unit, based on arelative position between the virtual display area and the seconddisplay unit.

According to this configuration, an image can be displayed by thedisplay device in correspondence to the position of the electronicdevice in the real space. An effect such as using this image tosupplement or expand the image displayed by the first display unit ofthe electronic device can be obtained.

Additionally, according to an aspect of the invention, the electronicdevice control unit is configured to initialize a display mode of theimage by using a position of the first display unit as a reference, whenthe electronic device control unit detects the first display unit beingpresent in the range where the first display unit is visually recognizedthrough the second display unit.

According to this configuration, the display mode can be adjusted incorrespondence to whether the first display unit of the electronicdevice can be recognized visually as the outside scene in the displaydevice.

Additionally, according to an aspect of the invention, the seconddisplay unit includes a display unit for a left eye configured to emitimaging light toward a left eye of a user wearing the display device,and a display unit for a right eye configured to emit imaging lighttoward the right eye of the user, and in correspondence to a position ofthe first display unit visually recognized as an outside scene by thesecond display unit, a display position by the display unit for a lefteye, and a display position by the display unit for a right eye arecontrolled, and a convergence angle of an image displayed by the seconddisplay unit is adjusted.

According to this configuration, the convergence angle of the imagedisplayed by the display device is adjusted, and accordingly, a distancein which the display image of the display device is visually recognizedcan be made correspond to the position of the first display unit of theelectronic device. Accordingly, the display by the display device andthe display by the electronic device can be coordinated moreappropriately.

Additionally, according to an aspect of the invention, the seconddisplay unit includes an optical system capable of adjusting a visualrecognition distance that the user perceives to an image displayed bythe second display unit, and the optical system is controlled incorrespondence to a convergence angle of an image displayed by thesecond display unit.

According to this configuration, since a distance in which the displayimage of the display device is visually recognized is made correspond tothe position of the first display unit of the electronic device, thedisplay by the display device and the display by the electronic devicecan be coordinated more appropriately.

Additionally, to address the above-described issue, an electronic deviceaccording to an aspect of the invention is an electronic deviceconnected with a display device having a head mounted type andconfigured to superimpose an image on an outside scene and to displaythe image, the electronic device including a first display unit, anoutput unit configured to output an image to the display device, and acontrol unit configured to, based on a relative position of the firstdisplay unit with respect to a second display unit provided in thedisplay device, cause the output unit to output an image used when thesecond display unit displays an image corresponding to a position of thefirst display unit visually recognized as an outside scene by thedisplay device.

According to this configuration, the electronic device connected withthe display device outputs the image to the display device, andaccordingly, the display device can perform display to correspond to theposition of the first display unit of the electronic device as theoutside scene.

Additionally, to address the above-described issue, a display methodaccording to an aspect of the invention is a display method using anelectronic device including a first display unit and a display devicehaving a head mounted type and including a second display unitconfigured to superimpose an image on an outside scene and to displaythe image, the display method including outputting an image by thedisplay device, and performing by the display device acquiring the imageoutput by the electronic device, causing the second display unit todisplay the acquired image, and causing the second display unit todisplay the image in correspondence to a position of the electronicdevice visually recognized as an outside scene in the second displayunit.

According to an aspect of the invention, when the image output by theelectronic device is displayed by the display device, the image isdisplayed to correspond to the position of the electronic device as theoutside scene. For example, since an image can be displayed by thedisplay device in correspondence to the position of the electronicdevice visible as the outside scene, display corresponding to thedisplay by the electronic device can be performed by the display device.

The aspects of the invention can also be achieved in various modes otherthan the display system, the electronic device, and the display methoddescribed above. For example, the aspects of the invention can beachieved in modes such as a program for executing the above-describeddisplay method by a computer, a recording medium storing the program, aserver device configured to deliver the program, a transmission mediumfor transmitting the program, and a data signal in which the program isembodied on a carrier wave.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is an appearance view of an HMD and a PC constituting a displaysystem.

FIG. 2 is a view illustrating a configuration of an optical system ofthe HMD.

FIG. 3 is a perspective view of a main portion of an image display unitas viewed from a head side of a user.

FIG. 4 is an explanatory view illustrating correspondence between adisplay unit of the HMD and the imaging range.

FIG. 5 is a block diagram of respective components constituting thedisplay system.

FIG. 6 is an explanatory view illustrating processing for determining arelative position of the PC with respect to the image display unit.

FIG. 7 is a flowchart illustrating an operation of the display system.

FIG. 8 is a view illustrating an example of a display mode of thedisplay system.

FIG. 9 is a view illustrating an example of a display mode of thedisplay system.

FIG. 10 is a view illustrating an example of a display mode of thedisplay system.

FIG. 11 is a flowchart illustrating an operation of the display system.

FIG. 12 is a view illustrating an example of a display mode of a displaysystem in a second exemplary embodiment.

FIG. 13 is a flowchart illustrating an operation of a display system ina third exemplary embodiment.

FIG. 14 is a flowchart illustrating an operation of the display systemin the third exemplary embodiment.

FIG. 15 is a block diagram of respective components constituting adisplay system in a fourth exemplary embodiment.

DESCRIPTION OF EXEMPLARY EMBODIMENTS 1. First Exemplary Embodiment

1-1. Configuration of Display System

FIG. 1 is a view illustrating a configuration of a display system 1according to a first exemplary embodiment to which the invention isapplied.

The display system 1 includes a Head Mounted Display (HMD: head mountedtype display device) 100, and a Personal Computer (PC) 300 as anexternal device of the HMD 100.

The HMD 100 is a display device including an image display unit 20(second display unit) configured to cause a user to visually recognize avirtual image in a state where the display device is worn on a head ofthe user, and a connecting device 10 configured to control the imagedisplay unit 20. The connecting device 10 includes a plurality ofconnectors 11 on a box-shaped case (may be referred to as a housing, ora body). The image display unit 20 and the connecting device 10 areconnected via a connecting cable 40.

In an example in FIG. 1, the connecting device 10 includes threeconnectors 11A, 11B, and 11C. In the following description, when theconnectors 11A, 11B, and 11C are not distinguished from one another, theconnectors 11A, 11B, and 11C are collectively referred to as a connector11. The connector 11 is a wired interface to be connected with acommunication cable, and via the communication cable, the connectingdevice 10 is connected with an external device. Each of the connectors11A, 11B, and 11C is, for example, a connector compliant with a knowncommunication interface standard, and may be a connector having anidentical shape, or may be a different type of connector. In theexemplary embodiment, as an example, the connector 11A is a connectorcompliant with an HDMI (trade name) (High Definition MultimediaInterface) standard. Additionally, the connector 11B is a UniversalSerial Bus (USB) Type-C connector. Additionally, the connector 11C is aMicroUSB connector.

In the example in FIG. 1, the connecting device 10 and the PC 300 areconnected via a cable 2. The cable 2 includes an HDMI cable 2Aconnecting the PC 300 and the connector 11A, and a USB cable 2Bconnecting the PC 300 and the connector 11B. In this example, the PC 300is connected with the connecting device 10 via an HDMI interface forimage transmission, and a USB interface for data communication.

The PC 300 is a computer including a display unit 330 configured todisplay an image, and corresponds to the electronic device of theinvention. The PC 300 is preferably a portable computer, and examples ofthe PC 300 include a tablet type computer, a note type computer, and asmartphone. The PC 300 in FIG. 1 includes the display unit 330 as thefirst display unit on a surface of a plate shaped body. The display unit330 includes a display panel 331 (FIG. 5) such as a liquid crystaldisplay panel, and an organic Electro Luminescent (EL) display panel,and a touch sensor 332 (FIG. 5) configured to detect a touch operationby a user is provided on a surface of the display panel 331.

The PC 300 is configured to function as external equipment with respectto the HMD 100. The external equipment may be any electronic device aslong as the electronic device includes a display screen and has afunction to display an image on the display screen, and the PC 300 ismerely described as an example in the exemplary embodiment.

The image display unit 20 is a mounted body to be worn on a head of auser and is a so-called head mounted type display (HMD). That is, theHMD 100 includes a configuration where the connecting device 10 to beconnected to an external device such as the PC 300 is connected with theimage display unit 20 as an HMD body. The image display unit 20 has aneyeglasses-like shape in the exemplary embodiment. The image displayunit 20 includes, in a body including a right holding part 21, a leftholding part 23, and a front frame 27, a right display unit 22 (displayunit for a right eye), a left display unit 24 (display unit for a lefteye), a right light-guiding plate 26, and a left light-guiding plate 28.

The right holding part 21 and the left holding part 23 extend rearwardfrom both ends of the front frame 27, respectively, and hold the imagedisplay unit 20 to a head of a user in a manner similar to temples of apair of eyeglasses. Here, one of both the ends of the front frame 27located on the right side of the user in a state where the user wearsthe image display unit 20 is referred to as an end ER, and the other endlocated on the left side of the user in a state where the user wears theimage display unit 20 is referred to as an end EL. The right holdingpart 21 extends from the end ER of the front frame 27 to a positioncorresponding to the right side of the head of the user in a state wherethe user wears the image display unit 20. The left holding part 23extends from the end EL to a position corresponding to the left side ofthe head of the user in a state where the user wears the image displayunit 20.

The right light-guiding plate 26 and the left light-guiding plate 28 areprovided on the front frame 27. The right light-guiding plate 26 islocated in front of the right eye of the user in a state where the userwears the image display unit 20, and causes the user to visuallyrecognize an image with the right eye. The left light-guiding plate 28is located in front of the left eye of the user in a state where theuser wears the image display unit 20, and causes the user to visuallyrecognize an image with the left eye.

The front frame 27 has a shape formed by coupling an end of the rightlight-guiding plate 26 and an end of the left light-guiding plate 28 toeach other, and this coupling position corresponds to a position betweeneyebrows of the user in a state where the user wears the image displayunit 20. The front frame 27 may include a nose pad provided in thecoupling position of the right light-guiding plate 26 and the leftlight-guiding plate 28, and configured to abut on a nose of the user ina state where the user wears the image display unit 20. In this case,the image display unit 20 can be held to the head of the user by thenose pad, the right holding part 21, and the left holding part 23.Additionally, a belt (not illustrated) configured to be in contact withto a back of the head of the user in a state where the user wears theimage display unit 20 may be coupled to the right holding part 21 andthe left holding part 23, and in this case, the image display unit 20can be held to the head of the user by the belt.

Each of the right display unit 22 and the left display unit 24 is amodule obtained by unitizing an optical unit and a peripheral circuit.

The right display unit 22 is a unit related to image display by theright light-guiding plate 26, and is provided on the right holding part21, and located near the right side of the head of the user in a statewhere the user wears the image display unit 20. The left display unit 24is a unit related to image display by the left light-guiding plate 28,and is provided on the left holding part 23, and located near the leftside of the head of the user in a state where the user wears the imagedisplay unit 20. Note that the right display unit 22 and the leftdisplay unit 24 are also collectively and simply referred to as a“display driving unit”.

The right light-guiding plate 26 and the left light-guiding plate 28 areoptical parts formed with a light transmissive resin or the like, andare configured to guide imaging light output by the right display unit22 and the left display unit 24 to the eyes of the user. The rightlight-guiding plate 26 and the left light-guiding plate 28 are, forexample, prisms.

A dimmer plate (not illustrated) may be provided on each of surfaces ofthe right light-guiding plate 26 and the left light-guiding plate 28.The dimmer plate is an optical element having a thin plate shape and atransmittance different according to a the wavelength range of light,and functions as a so-called wavelength filter. The dimmer plate is, forexample, disposed to cover a front side of the front frame 27 being anopposite side to a side of the eyes of the user. An optical property ofthis dimmer plate can be selected appropriately to adjust atransmittance of light in any wavelength range such as visible light,infrared light, and ultraviolet light, and to adjust a light amount ofoutside light entering the right light-guiding plate 26 and the leftlight-guiding plate 28 from an outside and passing through the rightlight-guiding plate 26 and the left light-guiding plate 28.

The image display unit 20 is configured to guide imaging light generatedby the right display unit 22 and the left display unit 24 to the rightlight-guiding plate 26 and the left light-guiding plate 28,respectively, and to use this imaging light to cause the user tovisually recognize a virtual image to display an image. When outsidelight passes through the right light-guiding plate 26 and the leftlight-guiding plate 28 and enters the eyes of the user from in front ofthe user, imaging light constituting the virtual image and the outsidelight enter the eyes of the user, and visibility of the virtual image isaffected by intensity of the outside light. Thus, for example, thedimmer plate can be mounted on the front frame 27, and the opticalproperty of the dimmer plate can be selected or adjusted appropriatelyto adjust easiness of visual recognition of the virtual image. In atypical example, the dimmer plate having a light transmittance to anextent that the user wearing the HMD 100 can visually recognize at leastan external scene can be used. Additionally, when the dimmer plate isused, an effect such as protecting the right light-guiding plate 26 andthe left light-guiding plate 28 to suppress a damage, adhesion of dustor the like on the right light-guiding plate 26 and the leftlight-guiding plate 28 can be expected. The dimmer plate may be provideddetachably on the front frame 27 or each of the right light-guidingplate 26 and the left light-guiding plate 28, or a plurality of types ofdimmer plates may be replaceably mounted, or the dimmer plate may beomitted.

Each of the right display unit 22 and the left display unit 24 of theimage display unit 20 is connected with the connecting device 10. In theHMD 100, the connecting cable 40 is connected with the left holding part23, and wiring connected with this connecting cable 40 is laid insidethe image display unit 20 to connect each of the right display unit 22and the left display unit 24 with the connecting device 10.

A camera 61 is disposed on the front frame 27 of the image display unit20. The camera 61 desirably captures an image in an outside scenedirection in which the user visually recognize an outside scene in astate where the user wears the image display unit 20, and the camera 61is provided at a position on a front face of the front frame 27 wherethe camera 61 does not block the outside light passing through the rightlight-guiding plate 26 and the left light-guiding plate 28. In theexample in FIG. 1, the camera 61 is disposed on the end ER side of thefront frame 27. The camera 61 may be disposed on the end EL side or maybe disposed at the coupling of the right light-guiding plate 26 and theleft light-guiding plate 28.

The camera 61 is a digital camera including an imaging element such as aCCD and a CMOS, an imaging lens and the like, and the camera 61according to the exemplary embodiment is a monocular camera, but mayinclude a stereo camera. The camera 61 is configured to capture an imageof at least a portion of an outside scene in a front side direction ofthe HMD 100, in other words, in a direction of a field of view of theuser in a state where the user wears the HMD 100. The outside scene canbe rephrased as a real space.

In other words, the camera 61 is configured to capture an image in therange or a direction superimposed on the field of view of the user, andis configured to capture an image in a direction in which the userfixates. A width of an angle of view of the camera 61 can be setappropriately, but in the exemplary embodiment, as described later, thewidth of an angle of view of the camera 61 includes an outside world theuser visually recognizes through the right light-guiding plate 26 andthe left light-guiding plate 28. More preferably, the imaging range ofthe camera 61 is set to enable capturing all the field of view of theuser that can be recognized visually through the right light-guidingplate 26 and the left light-guiding plate 28.

The HMD 100 includes a distance sensor 64. The distance sensor 64 isdisposed on a border portion of the right light-guiding plate 26 and theleft light-guiding plate 28. In a state where the user wears the imagedisplay unit 20, a position of the distance sensor 64 is nearly in themiddle of both the eyes of the user in a horizontal direction, and isabove both the eyes of the user in a vertical direction.

The distance sensor 64 is configured to detect a distance to an objectto be measured and located in a preset measurement direction. Themeasurement direction of the distance sensor 64 in the exemplaryembodiment is the front side direction of the HMD 100 and overlaps withan imaging direction of the camera 61.

FIG. 2 is a plan view of a main portion illustrating a configuration ofan optical system of the HMD 100. In FIG. 2, a left eye LE and a righteye RE of a user are illustrated for explanation.

As illustrated in FIG. 2, the right display unit 22 and the left displayunit 24 are disposed symmetrically on the right- and left-hand sides. Asa configuration where the right eye RE of the user is caused to visuallyrecognize an image, the right display unit 22 includes an OrganicLight-Emitting Diode (OLED) unit 221 configured to emit imaging light.Additionally, the right display unit 22 includes a right optical system251 including a lens group configured to guide imaging light L emittedby the OLED unit 221, and the like. The imaging light L is guided by theright optical system 251 to the right light-guiding plate 26.

The OLED unit 221 includes an OLED panel 223 and an OLED drive circuit225 configured to drive the OLED panel 223. The OLED panel 223 is aself-light emission type display panel including light-emitting elementsdisposed in a matrix and configured to emit light by organicelectro-luminescence to emit red (R) color light, green (G) color light,and blue (B) color light respectively. The OLED panel 223 has, as onepixel, a unit including one R element, one G element, and one B element,and includes a plurality of the pixels, and the OLED panel 223 forms animage with the plurality of pixels disposed in a matrix.

The OLED drive circuit 225 is configured to, based on image data inputfrom the connecting device 10, select and power the light-emittingelements provided in the OLED panel 223 to cause the light-emittingelements of the OLED panel 223 to emit light. The OLED drive circuit 225is fixed by bonding or the like to a rear face of the OLED panel 223,namely, a back side of a light-emitting surface of the OLED panel 223.The OLED drive circuit 225 may include, for example, a semiconductordevice configured to drive the OLED panel 223, and may be mounted on asubstrate (not illustrated) fixed to the rear face of the OLED panel223. A temperature sensor 217 is mounted on this substrate.

Note that the OLED panel 223 may include a configuration in whichlight-emitting elements configured to emit white color light aredisposed in a matrix, and color filters corresponding to the R color,the G color, and the B color respectively are disposed to besuperimposed on the light-emitting elements. Additionally, the OLEDpanel 223 of a WRGB configuration including light-emitting elementsconfigured to emit white (W) color light, in addition to thelight-emitting elements configured to emit the R color light, the Gcolor light, and the B color light respectively may be used.

The right optical system 251 includes a collimate lens configured tocollimate the imaging light L emitted from the OLED panel 223. Theimaging light L collimated by the collimate lens enters the rightlight-guiding plate 26. In an optical path configured to guide lightinside the right light-guiding plate 26, a plurality of reflective facesconfigured to reflect the imaging light L is formed. The imaging light Lis reflected multiple times inside the right light-guiding plate 26 andthen, is guided to the right eye RE side. In the right light-guidingplate 26, a half mirror 261 (reflective face) located in front of theright eye RE is formed. The imaging light L is reflected by the halfmirror 261 to be emitted from the right light-guiding plate 26 towardthe right eye RE, and this imaging light L forms an image on a retina ofthe right eye RE, and causes the user to visually recognize the image.

Additionally, as a configuration in which the left eye LE of the user iscaused to visually recognize an image, the left display unit 24 includesan OLED unit 241 configured to emit imaging light, and a left opticalsystem 252 including a lens group configured to guide the imaging lightL emitted by the OLED unit 241, and the like. The imaging light L isguided by the left optical system 252 to the left light-guiding plate28.

The OLED unit 241 includes an OLED panel 243, and an OLED drive circuit245 configured to drive the OLED panel 243. The OLED panel 243 is aself-light emission type display panel configured in a manner similar tothe OLED panel 223. The OLED drive circuit 245 is configured to, basedon image data input from the connecting device 10, select and powerlight-emitting elements provided in the OLED panel 243 to cause thelight-emitting elements of the OLED panel 243 to emit light. The OLEDdrive circuit 245 is fixed by bonding or the like to a rear face of theOLED panel 243, namely, a back side of a light-emitting surface of theOLED panel 243. The OLED drive circuit 245 may include, for example, asemiconductor device configured to drive the OLED panel 243, and may bemounted on a substrate (not illustrated) fixed to the rear face of theOLED panel 243. A temperature sensor 239 is mounted on this substrate.

The left optical system 252 includes a collimate lens configured tocollimate the imaging light L emitted from the OLED panel 243. Theimaging light L collimated by the collimate lens enters the leftlight-guiding plate 28. The left light-guiding plate 28 is an opticalelement in which a plurality of reflective faces configured to reflectthe imaging light L is formed, and the left light-guiding plate 28 is,for example, a prism. The imaging light L is reflected multiple timesinside the left light-guiding plate 28 and then, is guided to the lefteye LE side. In the left light-guiding plate 28, a half mirror 281(reflective face) located in front of the left eye LE is formed. Theimaging light L is reflected by the half mirror 281 to be emitted fromthe left light-guiding plate 28 to the left eye LE, and this imaginglight L forms an image on a retina of the left eye LE, and causes theuser to visually recognize the image.

The HMD 100 functions as a see-through type display device. Namely, theimaging light L reflected by the half mirror 261 and outside light OLhaving passed through the right light-guiding plate 26 enter the righteye RE of the user. Additionally, the imaging light L reflected by thehalf mirror 281 and the outside light OL having passed through the halfmirror 281 enter the left eye LE. Accordingly, the HMD 100 superimposesthe imaging light L of an image processed internally and the outsidelight OL on each other, and causes the imaging light L and the outsidelight OL superimposed on each other to enter the eyes of the user, andthe user views an outside scene through the right light-guiding plate 26and the left light-guiding plate 28, and visually recognizes the imageformed by the imaging light L and superimposed on this outside scene.Each of the half mirrors 261 and 281 is an image extracting unitconfigured to reflect imaging light output by each of the right displayunit 22 and the left display unit 24 and extract an image, and can bereferred to as a display unit.

Note that the left optical system 252 and the left light-guiding plate28 are collectively referred to as a “left light-guiding unit”, and theright optical system 251 and the right light-guiding plate 26 arecollectively referred to as a “right light-guiding unit”. Configurationsof the right light-guiding unit and the left light-guiding unit are notlimited to the example described above, and can use any manner as longas imaging light is used to form a virtual image in front of the eyes ofthe user. For example, a diffraction grating may be used, or asemi-transmissive reflection film may be used.

FIG. 3 is a perspective view of a main portion of the image display unit20 as viewed from the head side of the user, this is, a side being incontact with the head of the user of the image display unit 20, in otherwords, a side visible with the right eye RE and the left eye LE of theuser. In other words, back sides of the right light-guiding plate 26 andthe left light-guiding plate 28 are visible.

In FIG. 3, the half mirror 261 configured to irradiate the right eye REof the user with imaging light and the half mirror 281 configured toirradiate the left eye LE with imaging light are visible asapproximately square-shaped regions. Additionally, all the rightlight-guiding plate 26 and the left light-guiding plate 28 including thehalf mirrors 261 and 281 transmit the outside light as described above.Thus, the user visually recognizes an outside scene through all theright light-guiding plate 26 and the left light-guiding plate 28, andvisually recognizes rectangular display images at positions of the halfmirrors 261 and 281.

FIG. 4 is an explanatory view illustrating correspondence between theimage display unit 20 of the HMD 100 and the imaging range.

As described above, the camera 61 is disposed at the end on the rightside in the image display unit 20, and captures an image in a directionin which both the eyes of the user are directed, namely, in front of theuser.

FIG. 4 is a view schematically illustrating a position of the camera 61together with the right eye RE and the left eye LE of the user in a planview. An angle of view (imaging range) C of the camera 61 isillustrated. Note that the angle of view C in a horizontal direction isillustrated in FIG. 4, but an actual angle of view of the camera 61 alsoextends in a top-bottom direction as in a manner similar to a generaldigital camera.

An optical axis of the camera 61 extends in a direction including a lineof sight direction RD of the right eye RE and a line of sight directionLD of the left eye LE. The outside scene that can be recognized visuallyby the user in a state where the user wears the HMD 100 is notnecessarily an infinitely distant scene. For example, as illustrated inFIG. 4, when the user fixates an object OB with both the eyes, the linesof sight RD and LD of the user are directed to the object OB. In thiscase, a distance from the user to the object OB often ranges fromapproximately 30 cm to 10 m, and more often ranges from approximately 1m to 4 m. Thus, standards of an upper limit and a lower limit of thedistance from the user to the object OB in normal use may be defined forthe HMD 100. These standards may be determined by research or anexperiment, or may be set by the user. The optical axis and the angle ofview of the camera 61 may preferably be set such that when the distanceto the object OB in normal use corresponds to the set standard of theupper limit, and when the distance to the object OB in normal usecorresponds to the set standard of the lower limit, this object OB ispresent within the angle of view.

Additionally, in general, a visual field angle of a human isapproximately 200 degrees in the horizontal direction and approximately125 degrees in the vertical direction, and an effective field of viewexcellent in information acceptance performance of the visual fieldangle of a human is approximately 30 degrees in the horizontal directionand approximately 20 degrees in the vertical direction. Further, astable field of fixation in which a point of fixation at which a humanfixates is promptly and stably visible ranges from approximately 60degrees to 90 degrees in the horizontal direction, and ranges fromapproximately 45 degrees to 70 degrees in the vertical direction. Inthis case, when the point of fixation is located at the object OB inFIG. 4, the effective field of view is approximately 30 degrees in thehorizontal direction and approximately 20 degrees in the verticaldirection with the lines of sight RD and LD as the center. Additionally,the stable field of fixation ranges from approximately 60 degrees to 90degrees in the horizontal direction and from approximately 45 degrees to70 degrees in the vertical direction, and a visual field angle isapproximately 200 degrees in the horizontal direction, and approximately125 degrees in the vertical direction. Further, an actual visual fieldin which the user visually recognizes through the image display unit 20,and through the right light-guiding plate 26, and the left light-guidingplate 28 can be referred to as an actual Field Of View (FOV). In theconfiguration of the exemplary embodiment illustrated in each of FIG. 1and FIG. 2, the actual field of view corresponds to an actual visualfield in which the user visually recognizes through the rightlight-guiding plate 26 and the left light-guiding plate 28. The actualfield of view is narrower than the visual field angle and the stablefield of fixation, but wider than the effective field of view.

The angle of view C of the camera 61 can preferably capture an image inthe range wider than the visual field of the user, and specifically, theangle of view C is preferably wider than at least the effective field ofview of the user. Additionally, the angle of view C is more preferablywider than the actual field of view of the user. Further preferably, theangle of view C is wider than the stable field of fixation of the user,and most preferably, the angle of view C is wider than the visual fieldangles of both the eyes of the user.

The camera 61 may include a so-called wide angle lens as an imaging lensand may be configured to be capable of capturing an image at a wideangle of view. The wide angle lens may include lenses referred to as anultrawide lens and a quasi wide lens, or may be a single focus lens or azoom lens, or the camera 61 may be configured to include a lens groupincluding a plurality of lenses.

Additionally, as described above, the camera 61 in the exemplaryembodiment is disposed on the end ER side in the front frame 27 of theimage display unit 20, but may be disposed on the end EL side, or may bedisposed at the coupling of the right light-guiding plate 26 and theleft light-guiding plate 28. In this case, a position in a right-leftdirection of the camera 61 is different from the position in FIG. 4, andthe angle of view C is appropriately set according to the position ofthe camera 61. Specifically, when the camera 61 is on the end EL side,the angle of view C is directed diagonally forward right in FIG. 4.Additionally, for example, when the camera 61 is disposed at thecoupling of the right light-guiding plate 26 and the left light-guidingplate 28, the angle of view C is directed toward a front side of theimage display unit 20.

When the user views an object with the right eye RE and the left eye LE,the user perceives and recognizes a distance to the object according toan angle formed by the line of sight direction of the right eye RE andthe line of sight direction of the left eye LE. This angle is referredto as a convergence angle, and a convergence angle made when the objectOB illustrated in FIG. 4 is viewed is PA, for example.

A convergence angle made when the user views images displayed on thehalf mirrors 261 and 281 is an angle formed by a line of sight directionmade when the image on the half mirror 261 is viewed with the right eyeRE and a line of sight direction made when the image on the half mirror281 is viewed with the left eye LE. A degree of the convergence angle inthis case is determined by display positions of images on the halfmirrors 261 and 281. Therefore, a display position at which the rightdisplay unit 22 displays an image and a display position at which theleft display unit 24 displays an image are adjusted, and accordingly,the convergence angle can be controlled to control a sense of distancevisually recognized by the user. For example, with respect to the imagesdisplayed by the right display unit 22 and the left display unit 24, thesense of distance visually recognized by the user (visual recognitiondistance) can be adjusted.

Additionally, the distance sensor 64 is disposed to be directed forwardin the middle of the right light-guiding plate 26 and the leftlight-guiding plate 28.

1-2. Control System of Display System

FIG. 5 is a block diagram illustrating configurations of the HMD 100 andthe PC 300 constituting the display system 1.

As described above, the HMD 100 includes the connecting device 10 andthe image display unit 20 connected with each other via the connectingcable 40.

As described above, the image display unit 20 includes the right displayunit 22 and the left display unit 24. The right display unit 22 includesa display unit substrate 210. On the display unit substrate 210, aconnecting unit 211 configured to be connected with the connecting cable40, a reception unit (Rx) 213 configured to receive data input from theconnecting device 10 via the connecting unit 211, and an EEPROM 215(storage unit) are mounted.

The connecting unit 211 connects the reception unit 213, the EEPROM 215,the temperature sensor 217, the camera 61, the distance sensor 64, anilluminance sensor 65, and an LED indicator 67 with the connectingdevice 10.

The Electrically Erasable Programmable Read-Only Memory (EEPROM) 215stores various kinds of data in a non-volatile manner. The EEPROM 215stores, for example, data about light-emitting properties and displayproperties of the OLED units 221 and 241 provided in the image displayunit 20, and data about a property of a sensor provided in the rightdisplay unit 22 or the left display unit 24. Specifically, the EEPROM215 stores parameters regarding gamma correction of the OLED units 221and 241, data used to compensate for detection values of the temperaturesensors 217 and 239, and the like. These kinds of data are generated byinspection at the time of shipping of the HMD 100 from a factory, andare written into the EEPROM 215. The data stored in the EEPROM 215 canbe read by a control unit 120.

The camera 61 captures an image in accordance with a signal input viathe connecting unit 211 and outputs captured image data to theconnecting unit 211.

As illustrated in FIG. 1, the illuminance sensor 65 is provided at theend ER of the front frame 27 and is disposed to receive outside lightcoming from in front of the user wearing the image display unit 20. Theilluminance sensor 65 is configured to output a detection valuecorresponding to an amount of received light (intensity of receivedlight).

As illustrated in FIG. 1, the LED indicator 67 is disposed near thecamera 61 at the end ER of the front frame 27. The LED indicator 67 isconfigured to be turned on during image capturing by the camera 61 tonotify that the image capturing is in progress.

The temperature sensor 217 is configured to detect a temperature andoutput a voltage value or a resistance value corresponding to thedetected temperature as a detection value. The temperature sensor 217 ismounted on the rear face side of the OLED panel 223 (FIG. 3). Thetemperature sensor 217 may be mounted, for example, on the samesubstrate as the substrate on which the OLED drive circuit 225 ismounted. According to this configuration, the temperature sensor 217mainly detects a temperature of the OLED panel 223.

The distance sensor 64 is configured to execute distance detection, andoutput a signal indicating detection results to the connecting device 10via the connecting unit 211. As the distance sensor 64, for example, aninfrared ray type depth sensor, an ultrasonic type distance sensor, TimeOf Flight (TOF) type distance sensor, and a distance detection unitconfigured to perform image detection and audio detection in combinationcan be used. Additionally, the distance sensor 64 may be configured toprocess an image obtained in stereo photographing by a stereo camera ora monocular camera to detect a distance.

In FIG. 5, the single distance sensor 64 is illustrated, but a pair ofthe distance sensors 64 and 64 illustrated in FIG. 3 may operatesimultaneously. Additionally, each of the pair of distance sensors 64and 64 may be connected with the connecting unit 211 and operateindependently of each other.

The reception unit 213 is configured to receive image data fordisplaying transmitted from the connecting device 10 via the connectingunit 211, and output the image data to the OLED unit 221.

The left display unit 24 includes the display unit substrate 210. On thedisplay unit substrate 210, a connecting unit 231 configured to beconnected with the connecting cable 40, and a reception unit (Rx) 233configured to receive data input from the connecting device 10 via theconnecting unit 231 are mounted. Additionally, on the display unitsubstrate 210, a six-axis sensor 235 and a magnetic sensor 237 aremounted.

The connecting unit 231 connects the reception unit 233, the six-axissensor 235, the magnetic sensor 237, and the temperature sensor 239 withthe connecting device 10.

The six-axis sensor 235 is a motion sensor (inertial sensor) including athree-axis acceleration sensor and a three-axis gyro (angular velocity)sensor. As the six-axis sensor 235, an Inertial Measurement Unit (IMU)including the above-described sensors as modules may be adopted. Themagnetic sensor 237 is a three-axis geomagnetic sensor, for example.

The temperature sensor 239 is configured to detect a temperature andoutput a voltage value or a resistance value corresponding to thedetected temperature as a detection value. The temperature sensor 239 ismounted on the rear face side of the OLED panel 243 (FIG. 3). Thetemperature sensor 239 may be mounted, for example, on the samesubstrate as the substrate on which the OLED drive circuit 245 ismounted. According to this configuration, the temperature sensor 239mainly detects a temperature of the OLED panel 243.

Additionally, the temperature sensor 239 may be built in the OLED panel243 or the OLED drive circuit 245. Additionally, the substrate may be asemiconductor substrate. Specifically, when the OLED panel 243 ismounted, as an Si-OLED, together with the OLED drive circuit 245 and thelike to form an integrated circuit on an integrated semiconductor chip,the temperature sensor 239 may be mounted on this semiconductor chip.

Each component of the image display unit 20 operates with power suppliedfrom the connecting device 10 via the connecting cable 40. The imagedisplay unit 20 may include a power circuit (not illustrated) configuredto perform voltage conversion or distribution of the power supplied viathe connecting cable 40.

The connecting device 10 includes an I/F (interface) unit 110, thecontrol unit 120, a display controller 122, a sensor control unit 124, apower control unit 126, a non-volatile storage 130, an operating unit140, and a connecting unit 145. The I/F unit 110 as an acquisition unitincludes the connectors 11A, 11B, and 11C. Additionally, the I/F unit110 may include interface circuits (not illustrated) connected with theconnectors 11A, 11B, and 11C and configured to execute communicationprotocols compliant with respective communication standards.Additionally, the I/F unit 110 may be configured to receive power supplyvia the connectors 11A, 11B, and 11C.

The I/F unit 110, for example, may include an interface for a memorycard capable of being connected with an external storage device orstorage medium, or the like, or the I/F unit 110 may include a radiocommunication interface. The I/F unit 110 may be, for example, aninterface substrate on which the connectors 11A, 11B, and 11C, and aninterface circuit are mounted. Additionally, a configuration in whichthe control unit 120, the display controller 122, the sensor controlunit 124, and the power control unit 126 of the connecting device 10 aremounted on a connecting device main substrate (not illustrated) may beadopted. In this case, on the connecting device main substrate, theconnectors 11A, 11B, and 11C of the I/F unit 110, and an interfacecircuit may be mounted.

The control unit 120 is configured to control each component of theconnecting device 10. The control unit 120 includes a processor (notillustrated) such as a Central Processing Unit (CPU), and amicrocomputer. The control unit 120 causes the processor to execute aprogram to control each component of the HMD 100 in cooperation ofsoftware and hardware. Additionally, the control unit 120 may includeprogrammed hardware. The control unit 120 may include, together with theprocessor, a Random Access Memory (RAM) configured to form a work areaand a Read Only Memory (ROM) configured to store a control program.Additionally, the control unit 120 may be a semiconductor deviceincluding a processor, a RAM, and a ROM integrated.

The control unit 120 is connected with the non-volatile storage 130, theoperating unit 140, and the connecting unit 145.

The display controller 122 is configured to execute various kinds ofprocessing for the image display unit 20 to display an image based onimage data input to the I/F unit 110. For example, the displaycontroller 122 is configured to execute various kinds of processing suchas cutting out of a frame, resolution conversion (scaling), intermediateframe generation, and frame rate conversion. The display controller 122is configured to output image data corresponding to each of the OLEDunit 221 of the right display unit 22, and the OLED unit 241 of the leftdisplay unit 24 to the connecting unit 145. The image data input to theconnecting unit 145 is transmitted to the connecting units 211 and 231via the connecting cable 40.

As a specific example of the “cutting out” of a frame, the displaycontroller 122 lays out an entire image larger than a size of a displayarea in which the image display unit 20 displays an image, on an workarea. The display area is, for example, an area in which an image isformed in each of the OLED units 221 and 241. Additionally, a size of animage is indicated by the number of pixels, or resolution. The displaycontroller 122 transfers only data of the cut out area of the laid outimage to the right display unit 22 and the left display unit 24.

Additionally, the display controller 122 may be configured to acquireimage data larger than the display area of the image display unit 20,generate image data including only image data extracted from theacquired image data and having a size to be displayed by the imagedisplay unit 20, and transfer the generated image data to the rightdisplay unit 22 and the left display unit 24.

For example, when the image data input to the I/F unit 110 is 3D (threedimensional) image data, the display controller 122 is configured toexecute 3D image decode. In processing of the 3D image decode, thedisplay controller 122 is configured to generate a frame for the righteye and a frame for the left eye from the 3D image data. Examples of aformat of the 3D image data input to the I/F unit 110 include a side byside format, a top and bottom format, and a frame packing format, but 3Dmodel data may be used.

The display controller 122 is connected with the connector 11A and theconnector 11B provided in the I/F unit 110. The display controller 122is configured to execute processing on image data input to the connector11A and image data input to the connector 11B, as an object to beprocessed. Additionally, the display controller 122 may have a functionto transmit/receive various kinds of control data about transmission ofimage data to/from a device connected with the connector 11A or theconnector 11B.

The sensor control unit 124 is configured to control the camera 61, thedistance sensor 64, the illuminance sensor 65, the temperature sensor217, the six-axis sensor 235, the magnetic sensor 237, and thetemperature sensor 239. Specifically, the sensor control unit 124 isconfigured to set and initialize a sampling period of each sensoraccording to control by the control unit 120, and the sensor controlunit 124 is configured to, in correspondence to the sampling period ofeach sensor, power each sensor, transmit control data, and acquire adetection value, for example.

Additionally, the sensor control unit 124 is connected with theconnector 11B of the I/F unit 110, and is configured to output dataabout the detection value acquired from each sensor to the connector 11Bat preset timing. Accordingly, a device connected with the connector 11Bcan acquire the detection value of each sensor of the HMD 100, andcaptured image data of the camera 61. The data output by the sensorcontrol unit 124 may be digital data including the detection value.Additionally, the sensor control unit 124 may be configured to outputdata of results obtained by an arithmetic operation based on thedetection value of each sensor. For example, the sensor control unit 124is configured to integrally process detection values of a plurality ofsensors, and function as a so-called sensor fusion processing unit. Thesensor control unit 124 executes sensor fusion to output data determinedfrom the detection values of the sensors, for example, track data ofmovement of the image display unit 20, and relative coordinate data ofthe image display unit 20. The sensor control unit 124 may have afunction to transmit/receive various kinds of control data abouttransmission of data to/from a device connected with the connector 11B.

A processor such as a CPU may execute a program to achieve the displaycontroller 122 and/or the sensor control unit 124 in cooperation ofsoftware and hardware. Namely, each of the display controller 122 andthe sensor control unit 124 includes a processor, and executes a programto execute the above-described operations. In this example, a processorconstituting the control unit 120 may execute a program to achieve thedisplay controller 122 and the sensor control unit 124. In other words,the processor may be configured to execute the program to function asthe control unit 120, the display controller 122, and the sensor controlunit 124. Here, the processor can be rephrased as a computer.

Additionally, each of the display controller 122 and the sensor controlunit 124 may include programmed hardware such as a Digital SignalProcessor (DSP) and a Field Programmable Gate Array (FPGA).Additionally, the display controller 122 and the sensor control unit 124may be integrated to be constituted as a System-on-a-Chip (SoC)-FPGA.

The power control unit 126 is connected with the connector 11B and theconnector 11C provided in the I/F unit 110. The power control unit 126is configured to, based on power supplied from the connectors 11B and11C, supply power to each component of the connecting device 10 and theimage display unit 20. Additionally, the power control unit 126 mayinclude a voltage conversion circuit (not illustrated) built in, and maybe configured to be capable of supplying different voltage to eachcomponent of the connecting device 10 and the image display unit 20. Thepower control unit 126 may include a programmed semiconductor devicesuch as a logic circuit and the FPGA. Additionally, the power controlunit 126 may include hardware (including a processor) common to thedisplay controller 122 and/or the sensor control unit 124.

Each of the display controller 122, the sensor control unit 124 and thepower control unit 126 may include a work memory for performing dataprocessing, and may be configured to perform processing by using a workarea of a RAM (not illustrated) provided in the control unit 120.

The control unit 120 is configured to read data from the EEPROM 215provided in the right display unit 22, and the control unit 120 isconfigured to, based on the read data, set operations of the displaycontroller 122 and the sensor control unit 124. Additionally, thecontrol unit 120 is configured to, according to an operation in theoperating unit 140, cause respective components including the displaycontroller 122, the sensor control unit 124, and the power control unit126 to operate. Additionally, the control unit 120 is configured toidentify devices connected with the display controller 122, the sensorcontrol unit 124, and the power control unit 126 via the I/F unit 110,and control the display controller 122, the sensor control unit 124, andthe power control unit 126 to execute operations appropriate for therespective devices.

Additionally, the control unit 120 is configured to control start andstop of powering the LED indicator 67. For example, the control unit 120turns on or blinks the LED indicator 67 in correspondence to timing atwhich the camera 61 starts and ends capturing an image.

The non-volatile storage 130 is a storage device configured to storedata to be processed by the control unit 120, and the like in anon-volatile manner. The non-volatile storage 130 is, for example, amagnetic recording device such as a Hard Disk Drive (HDD), or is astorage device using a semiconductor storage element such as a flashmemory.

Additionally, the connecting device 10 may include a rechargeablebattery (not illustrated), and may be configured to supply power to eachcomponent of the connecting device 10 and the image display unit 20 fromthis battery.

The PC 300 includes a control unit 310, a non-volatile storage 320, thedisplay unit 330, an I/F (interface) unit 341, and a communication unit345. The control unit 310 (electronic device control unit) includes aprocessor (not illustrated) such as a CPU or a microcomputer, and thisprocessor is configured to execute a program to control each componentof the PC 300. The control unit 310 may include a ROM configured tostore, in a non-volatile manner, a control program to be executed by theprocessor (so-called computer), and a RAM constituting a work area ofthe processor.

The non-volatile storage 320 is configured to store, in a non-volatilemanner, a program to be executed by the control unit 310 and data to beprocessed by the control unit 310. The non-volatile storage 130 is, forexample, a magnetic recording device such as an HDD, or is a storagedevice using a semiconductor storage element such as a flash memory.

The non-volatile storage 320 is configured to store, for example,content data 321 of contents including an image. The content data 321 isa file in a format that the control unit 310 can process, and includesimage data, and may include audio data.

Additionally, the non-volatile storage 320 is configured to store anoperating system (OS) as a basic control program executed by the controlunit 310, an application program operating by using the OS as aplatform, and the like. Additionally, the non-volatile storage 320 isconfigured to store data processed during execution of the applicationprogram, data of processing results, and the like.

The display panel 331, and the touch sensor 332 provided in the displayunit 330 are connected with the control unit 310. The display panel 331is configured to display various images based on control of the controlunit 310. The touch sensor 332 is configured to detect a touch operationand output data indicating the detected operation to the control unit310. The data output by the touch sensor 332 is coordinate dataindicating an operating position in the touch sensor 332, or the like.

The I/F unit 341 is an interface connected with an external device, andcorresponds to the output unit according to the invention. The I/F unit341 is configured to execute communication compliant with, for example,a standard such as an HDMI interface and a USB interface. The I/F unit341 includes a connector (not illustrated) to be connected with a cable(for example, the cable 2), and an interface circuit (not illustrated)configured to process a signal transmitted via the connector. The I/Funit 341 is an interface substrate including the connector and theinterface circuit, and is connected with a main substrate on which aprocessor and the like of the control unit 310 are mounted.Alternatively, the connector and the interface circuit constituting theI/F unit 341 are mounted on a main substrate of the PC 300. In theexemplary embodiment, the I/F unit 341 includes the HDMI interface, andthe USB interface, and is connected with the connectors 11A and 11B viathe HDMI cable 2A and the USB cable 2B, respectively. For example, thecontrol unit 310 is configured to output image data via the HDMI cable2A, and receive data about an output value of a sensor and the like fromthe connecting device 10 via the USB cable 2B. The I/F unit 341 canexecute communication via the HDMI cable 2A and communication via theUSB cable 2B, independently. Additionally, the I/F unit 341 may be aradio communication interface. In this case, the I/F unit 341 can be aninterface substrate on which a communication circuit including an RFunit is mounted, or can be a circuit mounted on a main substrate.

The communication unit 345 is a communication interface configured toexecute data communication with an external device. The communicationunit 345 may be a wired communication interface capable of beingconnected with a cable, or may be a radio communication interface. Forexample, the communication unit 345 may be a wired LAN interfacesupporting Ethernet (trade name), or a wireless LAN interface supportingIEEE802.11 standards.

The control unit 310 is configured to execute the programs as describedabove to function as an input and output controller 311, a detectionvalue acquisition unit 312, a position detection unit 313, and an imageadjustment unit 315.

The input and output controller 311 is configured to, based on datainput from the touch sensor 332, detect input by the user. Additionally,the input and output controller 311 is configured to control input andoutput of data by the I/F unit 341 and the communication unit 345.

The detection value acquisition unit 312 is configured to acquire dataabout a detection value of each sensor provided in the HMD 100, from theconnecting device 10 connected via the I/F unit 341. The detection valueacquisition unit 312 is configured to acquire captured image data of thecamera 61, and data about detection values of the distance sensor 64,the illuminance sensor 65, the EEPROM 215, the six-axis sensor 235, themagnetic sensor 237, the temperature sensor 239, and the like, from theconnecting device 10. The data acquired by the detection valueacquisition unit 312 is data processed by the sensor control unit 124,and may be data including the detection value of each sensor, or datastatistically processed by the sensor control unit 124.

The position detection unit 313 is configured to, based on the dataacquired by the detection value acquisition unit 312, detect a positionof the HMD 100. More specifically, the position detection unit 313 isconfigured to detect a relative position between the PC 300 and theimage display unit 20. Here, the relative position detected by theposition detection unit 313 may be a position in a space in which theimage display unit 20 and the PC 300 exist, or may include a relativedirection between the image display unit 20 and the display unit 330.For example, the relative position detected by the position detectionunit 313 may be information indicating the position and/or the directionof the display unit 330 with respect to the image display unit 20.Additionally, for example, the relative position detected by theposition detection unit 313 may be information indicating the positionand/or the direction of the image display unit 20 with respect to the PC300. Additionally, for example, the relative position detected by theposition detection unit 313 may include coordinates in a threedimensional coordinate system set in the space in which the imagedisplay unit 20 and the PC 300 exist.

The position detection by the position detection unit 313 will bedescribed later.

The image data PC 300 outputs to the connecting device 10 can include,in addition to image data obtained by playing the content data 321,image data in a screen the PC 300 causes the display unit 330 todisplay. In this case, the connecting device 10 is configured to displayan identical screen to a screen the display unit 330 display, andperform so-called mirroring display.

Additionally, the image adjustment unit 315 may be configured to causethe display unit 330 to display a larger image including an image theconnecting device 10 is displaying and an image not displayed by the PC300. Additionally, for example, the connecting device 10 may beconfigured to expand and display a portion of the image the PC 300displays. A display mode in the PC 300 will be described later.

1-3. Operation of Display System

FIG. 6 is an explanatory view illustrating processing for determiningthe relative position of the PC 300 with respect to the image displayunit 20, and particularly illustrates calibration processing. FIG. 7 isa flowchart illustrating an operation of the display system 1, andparticularly illustrates an operation of the PC 300 in the calibrationprocessing.

In FIG. 6, a visual field VR a user visually recognizes through theimage display unit 20, and an imaging range VC of the camera 61 areillustrated. In this example, the imaging range VC overlaps with thevisual field VR but does not coincide with the visual field VR. Theimaging range VC and the visual field VR may coincide with each other,or the imaging range VC may be present in the visual field VR.

The display system 1 is used to execute, as illustrated in FIG. 6,calibration in a state where the PC 300 exists in the imaging range VCof the camera 61.

In the exemplary embodiment, a reference point P1 is set on the displayunit 330, and a reference position P2 is set on the image display unit20. The reference point P1 is display a user can visually recognize, andis a so-called marker. The PC 300 can detect the reference point P1 fromthe captured image data of the camera 61, by image processing. Thereference position P2 is set as a reference of a position in the imagedisplay unit 20, and may be a virtual position, or there may be nodisplay nor object indicating the reference position P2.

The reference point P1 may be any object or position explicitly andvisually recognized, and it is not necessary to newly provide display oran object to be the reference point P1. A specific position of a patternor a shape on the display unit 330 such as one of four corners of thedisplay unit 330 may be used as the reference point P1.

FIG. 7 is a flowchart illustrating an operation of the display system 1,and illustrates an operation of the PC 300 performing the calibration.

In the calibration, a user moves the head or the image display unit 20to locate the reference point P1 at a preset position in the visualfield VR of the user (step S11). When the position set in the visualfield VR and the reference point P1 are superimposed on each other, theuser performs an operation for confirming the position on the imagedisplay unit 20 or the PC 300 (step S12). For example, the user performsa knocking operation on the front frame 27 of the image display unit 20.This operation can be detected as acceleration by the six-axis sensor235. Additionally, when the image display unit 20 is equipped with aknock sensor (not illustrated), such an operation can be detected basedon a detection value of the knock sensor. The operation at step S12 maybe an operation including detection by the touch sensor 332.Additionally, at step S12, the PC 300 may detect an operation by audio.Namely, in a configuration in which the PC 300 is equipped with amicrophone, and the control unit 310 can process audio collected by themicrophone, the control unit 310 can execute audio recognitionprocessing for converting audio into a text, or audio command detectionprocessing for detecting audio of a pattern registered in advance. Inthis configuration, the control unit 310 may detect audio produced bythe user as an input operation at step S12.

The control unit 310 acquires captured image data of the camera 61generated when the operation at step S12 is detected, and determines aposition of the reference point P1 in the captured image data of thecamera 61 (step S13). Accordingly, the position in the captured imagedata of the camera 61 is associated with the position in the visualfield VR of the user wearing the image display unit 20. Therefore, whenthe PC 300 is imaged in the captured image data of the camera 61, arelative position of the PC 300 with respect to the visual field VR canbe specified from this captured image data.

The control unit 310 acquires data about output values of the six-axissensor 235 and the magnetic sensor 237 (step S14). The output values ofthe sensors acquired at step S14 indicate a reference position of theimage display unit 20 in a state where the reference point P1 issuperimposed on the set position in the visual field VR. For example, adirection and an amount of movement of the image display unit 20 fromthe reference position can be determined by integrating the outputvalues of the six-axis sensor 235. Additionally, based on the outputvalues of the magnetic sensor 237, a relative direction of the imagedisplay unit 20 with respect to the reference position of the imagedisplay unit 20 can be determined. Therefore, a relative positionbetween the image display unit 20 and the PC 300 can be determined.

The control unit 310 generates calibration data including the outputvalues acquired at step S14, stores the calibration data in thenon-volatile storage 320 (step S15) and ends this processing.

For example, steps S13 and the S15 are executed by the positiondetection unit 313, and step S14 is executed by the detection valueacquisition unit 312.

In the calibration processing illustrated in FIG. 7, the reference pointP1 of the PC 300 and the set position in the visual field VR are madecorrespond to each other, and relative position relation of the imagedisplay unit with respect to the reference position of the image displayunit is associated. The calibration data generated in this processingis, for example, data indicating a reference of a relative positionbetween the reference point P1 and the visual field VR. In thiscalibration, in addition to superimposing the predetermined position inthe visual field VR on the reference point P1, a technique in which thePC 300 is disposed in a preset distance from the image display unit 20may be adopted at step S11. For example, at step S11, an image used forguiding placement of the PC 300 at a position separated by apredetermined distance from the image display unit 20 may be displayedby the image display unit 20. This image can be used as a marker such asa straight line, a rectangle, and a point indicating a size of the PC300 visually recognized when the PC 300 is located at the positionseparated by the predetermined distance. The user may adjust a relativeposition between the PC 300 and the user to cause the PC 300superimposed on the marker of the image display unit 20 to be visible,and subsequently, may make the reference point P1 correspond to thepredetermined position to perform the operation at step S12.

Each of FIG. 8, FIG. 9, and FIG. 10 is a view illustrating an example ofa display mode of the display system 1 in the exemplary embodiment. Ineach of FIG. 8, FIG. 9, and FIG. 10, a visual field VR in which the userwearing the image display unit 20 can visually recognize through theimage display unit 20 is illustrated. Additionally, in the exampleillustrated in each of these figures, a size and a position of a displayarea in which the image display unit 20 displays an image to enable theuser to visually recognize are identical to a size and a position of thevisual field VR, and the display area and the visual field VR overlapwith each other. The invention is applicable to a configuration in whicha display area of the image display unit 20 is smaller than the visualfield VR.

In FIG. 8, a virtual image VD generated by the PC 300 is illustrated.The virtual image VD includes an area VD1 including an image identicalto an image the PC 300 causes the display unit 330 to display, and areasVD2, VD3, and VD4 adjacent to the area VD1. That is, the virtual imageVD is an image having a size larger than a size of the image displayedby the display unit 330.

The PC 300 is configured to set a virtual display area larger than adisplay area of the display panel 331, and generate the virtual image VDas an image to be displayed in this display area. The virtual displayarea and the virtual image VD may differ in resolution or size, but inthe exemplary embodiment, the virtual display area and the virtual imageVD are described to coincide with each other in resolution or size.

In the following description, a size of each of the image displayed bythe display unit 330, the virtual image VD, and the areas VD1, VD2, VD3,and VD4 is indicated by display resolution. For example, when displayresolution of the display unit 330 is 1920 dots horizontally×1080 dotsvertically and resolution of the virtual image VD and the virtualdisplay area is 3840 dots×2160 dots, a size of the virtual image VD isfour times the display image of the display unit 330.

Aspect ratios of the image displayed by the display unit 330 and thevirtual image VD may differ from each other, but when the aspect ratioscoincide with each other, there is an advantage that processingperformed when the virtual image VD including the display image of thedisplay unit 330 is generated can be executed rapidly. Additionally,there is an advantage that when the user visually recognizes both thedisplay image of the display unit 330 and the virtual image VD, there isno uncomfortable feeling. Thus, the case where the aspect ratioscoincide with each other is preferable. For a similar reason, the aspectratios and the resolution of the respective areas VD1 to VD4constituting the virtual image VD may not coincide with the aspect ratioand the resolution of the display unit 330, but preferably coincide withthe aspect ratio and the resolution of the display unit 330.Additionally, the resolution of the virtual image VD may not be aninteger multiple of the display resolution of the display unit 330, butis preferably an integer multiple.

The size of the virtual image VD is preferably identical to or largerthan a size of an image that the image display unit 20 can display.Namely, the resolution of the virtual image VD is preferably equal to orgreater than the display resolution of the display area of the imagedisplay unit 20. An example includes a configuration in which thedisplay resolution of the display unit 330 is 1920 dotshorizontally×1080 dots vertically, and the display resolution of theimage display unit 20 is 3840 dots×1080 dots, and the resolution of thevirtual image VD is 3840 dots×2160 dots. Additionally, the aspect ratiosof the virtual image VD and the display area of the image display unit20 may differ from each other, and the aspect ratios and the resolutionof the respective areas VD1 to VD4 may not coincide with the aspectratio and the resolution of the display area of the image display unit20. Additionally, the resolution of the virtual image VD may not be aninteger multiple of the resolution of the display area of the imagedisplay unit 20.

It can be said that the virtual image VD is an expanded display areaobtained by expanding the image the PC 300 causes the display unit 330to display. For example, when the display unit 330 displays a screen foroperation as a user interface of an operating system or an applicationprogram executed by the PC 300, the control unit 310 constitutes thescreen for operation in correspondence to the display resolution of thedisplay unit 330. When the virtual image VD is used, the control unit310 constitutes the screen as the user interface in correspondence tothe resolution of the virtual image VD, and causes the display unit 330to display a portion of the screen. The same applies to a case where thePC 300 plays the content data 321, or a case where other images or thelike are displayed. Accordingly, the virtual image VD includes the imagedisplayed by the display unit 330. In other words, the display unit 330displays a portion of the virtual image VD.

Additionally, the size of each of the virtual image VD, the displayimage of the display unit 330, and the image displayed by the imagedisplay unit 20 may be defined as a size in which the user wearing theimage display unit 20 visually recognizes.

A size of an image is determined by a size of a pixel and resolution. Itis appropriate that the size of the pixel of the display unit 330 beconsidered as a size of a pixel obtained when the user visuallyrecognizes the display unit 330 through the image display unit 20.Additionally, since the virtual image VD is not visually recognizedunless the image display unit 20 displays the virtual image VD, it isappropriate that the size of the pixel of the virtual image VD beconsidered as a size obtained when the user visually recognizes a pixelof the image displayed by the image display unit 20. Then, when a sizein which the user wearing the image display unit 20 visually recognizesis a reference, the virtual image VD may be larger than the display unit330. In this case, the resolution of the virtual image VD may be equalto or less than the display resolution of the display unit 330. Further,resolution of image data cut out of the virtual image VD and output tothe image display unit 20 may be resolution equal to or less than thedisplay resolution of the display unit 330. The control unit 310 isconfigured to perform resolution conversion for converting the displayimage of the display unit 330 to an image having lower resolution, inprocessing for generating the virtual image VD including the displayimage of the display unit 330, and add another image around the imageobtained after the conversion to generate the virtual image VD.

FIG. 11 is a flowchart illustrating an operation related to display ofthe display system 1, and particularly illustrates an operation of thePC 300. In the exemplary embodiment, the HMD 100 is configured todisplay an image based on image data output by the PC 300 via the HDMIcable 2A, and the PC 300 is configured to perform display control basedon the relative position between the image display unit 20 and the PC300.

Hereinafter, with reference to FIGS. 8 to 10 together with FIG. 11, theoperation of the display system 1 will be described.

The operation illustrated in FIG. 11 is an operation in which thecontrol unit 310 performs display using the virtual image VD. Inresponse to an operation detected by the touch sensor 332, the controlunit 310 instructs start of image output to the HMD 100, and when adisplay mode using the virtual image VD is specified, the control unit310 executes the operation in FIG. 11.

In the following description, a case where the display unit 330 ispresent in a visual field of the image display unit 20 refers to a casewhere the user wearing the HMD 100 can visually recognize through theimage display unit 20 the display unit 330 of the PC 300 as an outsidescene. Visual fields of the right eye RE and the left eye LE through theimage display unit 20, namely, the range of the outside scene visible tothe user can be determined from a direction of the image display unit20. Additionally, as illustrated in FIG. 4, in a case where relationbetween the angle of view of the camera 61, and the visual fields of theright eye RE and the left eye LE obtained when the outside scene isviewed through the image display unit 20 is known, the control unit 310can determine from the captured image data of the camera 61 whether thedisplay unit 330 is in the visual.

The control unit 310 acquires the captured image data of the camera 61and the output value of each sensor from the connecting device 10 viathe USB cable 2B (step S21). At step S21, the control unit 310 acquiresat least the captured image data, and the output value of the distancesensor 64. Step S21 is executed by, for example, the detection valueacquisition unit 312.

The control unit 310 determines whether the display unit 330 is presentin the captured image data of the camera 61 (step S22). At step S22, thecontrol unit 310 detects, for example, an image of the display unit 330from the captured image data by pattern matching.

In a case where the display unit 330 is determined not to be present inthe captured image data (step S22; NO), the control unit 310 returns tostep S21.

In a case where the display unit 330 is determined to be present in thecaptured image data (step S22; YES), the control unit 310 detects, basedon the output value of each sensor acquired at step S21, a position andan attitude of the display unit 330 (step S23). At step S23, the controlunit 310 determines, for example, based on execution of SimultaneousLocalization and Mapping (SLAM) or the detection value of the distancesensor 64, a relative position between the image display unit 20 and thedisplay unit 330. Additionally, the control unit 310 may be configuredto detect the reference point P1 of the display unit 330 by using thecaptured image data, and determine from the position of the referencepoint P1 detected the relative position of the display unit 330 withrespect to the image display unit 20.

The control unit 310 maps, according to the position and the attitudedetermined at step S23, the position and the attitude of the displayunit 330 in a three dimensional space (step S24). The position and theattitude of the display unit 330 are expressed by three dimensionalcoordinates with the reference position P2 being an origin or areference point.

The control unit 310 disposes the virtual image VD in a threedimensional space, in correspondence to the position and the directionof the display unit 330 (step S25). A size of the virtual image VD ispreset. For example, as set data (not illustrated), data defining theresolution of the virtual image VD or the position of the virtual imageVD to be set with respect to the display unit 330 is stored in thenon-volatile storage 320.

The control unit 310 disposes the virtual image VD to be parallel to thedisplay unit 330, and to be a face superimposed on the display unit 330.In other words, the virtual image VD includes the display unit 330, andis a virtual plane larger than the display unit 330. The control unit310 determines three dimensional coordinates with the reference positionP2 being an origin or a reference point, as for a reference positionbased on which a position of the virtual image VD such as positions offour corners or a position of the center of the virtual image VD can bespecified.

The control unit 310 generates image data of the virtual image VD (stepS26). The virtual image VD is a screen obtained by virtually expandingthe display unit 330 as described above, and, for example, is a screenconstituting a user interface of an operating system.

The control unit 310 determines the range to be displayed in the virtualimage VD by the image display unit 20 (step S27). As illustrated in FIG.8, the image display unit 20 displays a portion of the virtual image VD.The control unit 310 determines the range to be displayed in the virtualimage VD by the image display unit 20 from a direction of the imagedisplay unit 20 and a relative position with respect to the virtualimage VD.

The control unit 310 specifies a position of the display unit 330 in therange in which the image display unit 20 displays an image (step S28).The image display unit 20 displays an image on the half mirrors 261 and281, and the range in which the image is visually recognized issuperimposed on the visual field VR in the examples in FIG. 8 to FIG.10. When the display unit 330 is visually recognized through the imagedisplay unit 20 in the visual field VR of the user wearing the imagedisplay unit 20, the control unit 310 specifies a position at which thedisplay unit 330 is visible in the visual field VR. This position can bedetermined from the position and the attitude of the PC 300 specified atstep S23.

The control unit 310 performs processing for cutting out or extractingthe image present in the range to be displayed by the image display unit20 from the virtual image VD (step S29), and applies masking processingto the cut out image (step S30). The range in which the maskingprocessing is applied at step S30 is the range in which the display unit330 is superimposed on the image to be displayed by the image displayunit 20, or the range including a portion superimposed on the displayunit 330.

In the example illustrated in FIG. 8, the areas VD1 and VD2 of thevirtual image VD are cut out as the range to be displayed by the imagedisplay unit 20. The display unit 330 is visually recognized at aposition at which the area VD1 is displayed. In this case, the controlunit 310 may apply the masking processing to all the area VD1, or mayapply the masking processing only to the range in which the area VD1 issuperimposed on the display unit 330.

The user can visually recognize an outside scene passing through theimage display unit 20 in a state where the image display unit 20displays the virtual image VD. Here, when the display unit 330 ispresent in the visual field VR that the image display unit 20 visuallyrecognizes as the outside scene, an image displayed by the display unit330 and an image displayed by the image display unit 20 superimposed oneach other are visible for the user. To provide a better visualexperience to the user in the above-described state, in the exemplaryembodiment, processing for adjusting a display mode of the image displayunit 20 to enhance visibility of the display unit 330 is performed. Thisprocessing is referred to as the masking processing.

FIG. 9 illustrates a state in which the control unit 310 applies themasking processing. In an example in FIG. 9, the masking processing isapplied to all the area VD1 being the range superimposed on the displayunit 330. The masking processing is, for example, processing for turningcolor data of a pixel (so-called pixel data, a pixel value) into a blackcolor or a dark color. The image display unit 20 displays an image bycausing light emitted by the OLED units 221 and 241 to enter the righteye RE and the left eye LE, and thus when the pixel value turns into theblack color or the dark color, amounts of light entering the right eyeRE and the left eye LE decrease. When an amount of light emitted by theimage display unit 20 decreases, an amount of light of the outside lightOL passing through the image display unit 20 relatively increases, andthus visibility of the outside scene is enhanced. Therefore, the usercan visually recognize the outside scene favorably in the range in whichthe masking processing is applied, and the visibility of the displayunit 330 as the outside scene is enhanced by the masking processing.

Accordingly, the masking processing performed by the control unit 310 isprocessing in which a pixel value of image data output to the connectingdevice 10 to cause the image display unit 20 to display is set to theblack color (R, G, B=0, 0, 0) or a preset dark color to reduce an amountof light.

Since the pixel value in the range including the image displayed by thedisplay unit 330 is changed by the masking processing to the black coloror to the dark color in the image data that the PC 300 outputs to theconnecting device 10, the image displayed by the display unit 330 issubstantially removed. Accordingly, when the range in which the imagedisplay unit 20 displays an image is superimposed on the display unit330, the control unit 310 outputs to the connecting device 10 the imagedata obtained by removing the image displayed by the display unit 330.

In the image data subjected to the masking processing, the image in thearea VD1 being the area superimposed on the display unit 330 is removedby a mask, and an image in an area located around the removed image (thearea VD2 in FIG. 10) is displayed by the image display unit 20.

The control unit 310 sets a convergence angle of the image subjected tothe masking processing (step S31). At step S31, the control unit 310sets the convergence angle of the image such that a portion of thevirtual image VD to be displayed by the image display unit 20 isvisually recognized as a plane including the display unit 330. The PC300 outputs image data including image data of an image to be displayedby the right display unit 22 of the image display unit 20, and imagedata of an image to be displayed by the left display unit 24. Thecontrol unit 310 adjusts a position of the virtual image VD in the imagefor the right display unit 22, and a position of the virtual image VD inthe image for the left display unit 24 to adjust a convergence anglebetween the left eye LE and the right eye RE of the user.

The control unit 310 starts outputting the image data to the connectingdevice 10 (step S32). The connecting device 10 starts a processing inwhich the image data input to the I/F unit 110 is decoded by the displaycontroller 122, and is displayed by the right display unit 22 and theleft display unit 24.

Steps S24 to S32 are, for example, executed by the image adjustment unit315.

After the start of the display, the control unit 310 determines whetherthe visual field VR has moved (step S33). Here, the movement of thevisual field VR means movement of a relative position of the visualfield VR with respect to the virtual image VD. At step S33, the imageadjustment unit 315 performs the determination based on a detectionvalue of the position detection unit 313. Additionally, the followingsteps S34 to S39 are executed by the image adjustment unit 315.

FIG. 10 illustrates an example in which the visual field VR has moved.In the example in FIG. 10, the visual field VR has moved upward ascompared to the state in FIG. 9. Owing to this movement, the visualfield VR is superimposed on the areas VD1, VD2, VD3, and VD4 to crossthe areas VD1, VD2, VD3, and VD4.

When the control unit 310 determines that the visual field VR has moved(step S33; YES), the control unit 310 determines whether a position ofthe PC 300, more specifically a position of the display unit 330 is outof the range in which the image display unit 20 displays an image (stepS34). At step S34, the control unit 310 performs the determination basedon a relative position and a relative direction between the imagedisplay unit 20 and the display unit 330.

When the control unit 310 determines that the position of the displayunit 330 is out of the display range of the image display unit 20 (stepS34; YES), the control unit 310 cancels the masking processing appliedat step S30 (step S35). The control unit 310 changes, in correspondenceto the visual field VR having moved, the range to be cut out of thevirtual image VD (step S36), and returns to step S33.

In the example illustrated in FIG. 10, the control unit 310 cuts therange crossing the areas VD1, VD2, VD3, and VD4 out of the virtual imageVD, generates image data corresponding to the cut out image, and outputsthe generated image data to the connecting device 10. Accordingly, theimage display unit 20 displays the range in which the virtual image VDis superimposed on the visual field VR.

Additionally, when the control unit 310 determines that the position ofthe display unit 330 is not out of the display range of the imagedisplay unit 20 (step S34; NO), the display unit 330 is within thedisplay range of the image display unit 20. In this case, the controlunit 310 determines whether the display unit 330 has moved from anoutside of the display range of the image display unit 20 into thedisplay range (step S37).

In a case where, owing to the movement of the visual field VR, thedisplay unit 330 enters the display range from the state where thedisplay unit 330 has been out of the display range of the image displayunit 20 (step S37; YES), the control unit 310 initializes a state ofimage data to be output to the connecting device 10 (step S38). Duringthe initialization, the control unit 310 resets application/cancellationof the masking processing, the range to be cut out of the virtual imageVD, or the like in the image data output to the connecting device 10 toan initial state. The control unit 310 returns to step S23, and executesagain the processing for starting output of the image data.

Specific examples of the initialization processing at step S38 includesthe following examples. A first example includes processing in whichwhen it is determined that all the display unit 330 has entered thevisual field VR at step S37, the virtual image VD disposed and processedat step S25 previously executed is used to generate a display image.Additionally, second processing includes processing for executing theprocessing at step S23 and the subsequent steps in correspondence to theposition of the display unit 330, and newly resetting a position of thevirtual image VD (including a depth). In addition to these kinds ofprocessing, a specific example of the processing performed by thecontrol unit 310 is arbitrary.

Additionally, in a case where the display unit 330 has not moved fromthe outside of the display range of the image display unit 20 into thedisplay range (step S37; NO), namely, in a case where the state in whichthe display unit 330 is in the display range of the image display unit20 is continued, the control unit 310 shifts to step S36.

Additionally, when the control unit 310 determines that the visual fieldVR has not moved (step S33; NO), the control unit 310 determines whetherto end the display (step S39). For example, when the display end isinstructed by an operation on the display unit 330 or the like (stepS39; YES), the control unit 310 stops outputting the image data to theconnecting device 10 and ends this processing. On the other hand, whenthe display is to be continued (step S39; NO), the control unit 310returns to step S33.

At step S33, in a case where the image display unit 20 has moved, forexample, in a case where the user wearing the image display unit 20 hasmoved the head, the control unit 310 determines that the visual field VRhas moved. Additionally, in a case where the PC 300 has moved in a statewhere the image display unit 20 has not moved, and accordingly, arelative position between the image display unit 20 and the PC 300 haschanged, it may also be determined that the visual field VR has moved.That is, the movement of the visual field VR detected at step S33 is achange in the relative position between the image display unit 20 andthe PC 300, and includes a case where the image display unit 20 moves, acase where the PC 300 moves, and a case where both the image displayunit 20 and the PC 300 move.

Therefore, in any of a case where the user has moved the head, and acase where the user or another person has moved the PC 300, theprocessing at step S34 and the subsequent steps is executed depending onwhether the position of the PC 300 is within the display range of theimage display unit 20.

In the processing illustrated in FIG. 7 and FIG. 11, the display system1 may use the captured image data of the camera 61, instead of thedistance sensor 64, to detect a distance. For example, the displaysystem 1 stores in advance as reference data in the non-volatile storage130 or the non-volatile storage 320 data for associating a distance fromthe image display unit 20 to an object to be captured with a size of animage of an object to be captured and imaged in the captured image dataof the camera 61. In this case, the control unit 310 can calculate adistance from the image display unit 20 to the PC 300 or to the displayunit 330, based on a size of an image of the PC 300 or the display unit330 in the captured image data of the camera 61, and the reference data.

As described above, the display system 1 in the first exemplaryembodiment includes the PC 300 including the display unit 330, and theHMD 100 connected with the PC 300. The PC 300 includes the I/F unit 341configured to output an image. The HMD 100 includes the I/F unit 110configured to acquire the image data output by the PC 300. Additionally,the HMD 100 includes the image display unit 20 configured to enable anoutside scene to be visually recognized, superimpose an image on theoutside scene visually recognized in a state where the HMD 100 is worn,and display the image, and the display controller 122 configured tocause the image display unit 20 to display the image acquired by theacquisition unit. The display controller 122 is configured to cause theimage display unit 20 to display the image in correspondence to theposition of the PC 300 visually recognized as the outside scene.

Additionally, the PC 300 corresponds to the electronic device of theinvention connected with the HMD 100. The PC 300 includes the displayunit 330, and the I/F unit 341 configured to output an image data to theHMD 100. Additionally, the PC 300 includes the control unit 310configured to determine a relative position of the display unit 330 withrespect to the image display unit 20 of the HMD 100. The control unit310 is configured to generate, based on the relative position of thedisplay unit 330 with respect to the image display unit 20, image dataused for the image display unit 20 to display an image corresponding tothe position of the display unit 330 visually recognized as the outsidescene, and output the generated image data from the I/F unit 341.

According to the display system 1 and the PC 300 to which the inventionis applied, an image output by the PC 300 can be displayed by the HMD100 in correspondence to the position of the display unit 330 of the PC300 as the outside scene. For example, the image can be displayed by theHMD 100 in correspondence to the position of the display unit 330 of thePC 300 visible as the outside scene. Accordingly, the displaycorresponding to the display unit 330 of the PC 300 visually recognizedas the outside scene can be performed by the HMD 100 configured tosuperimpose the image on the outside scene and display the image.

Additionally, the control unit 310 is configured to generate the virtualimage VD corresponding to a virtual display area wider than the displayunit 330. The control unit 310 is configured to cause the display unit330 to display a portion of the virtual image VD generated incorrespondence to the virtual display area, and cause the I/F unit 341to output an image for display including at least a portion of thevirtual image VD. For example, the control unit 310 is configured tocause the display unit 330 to display a portion of the virtual image VD,and cause the I/F unit 341 to output an image obtained by removing aportion displayed by the display unit 330 from the virtual image VD.Additionally, the display system 1 is configured to cause the imagedisplay unit 20 provided in the HMD 100 to display an image based onimage data output by control of the control unit 310, in correspondenceto the position of the display unit 330 visually recognized as theoutside scene. Specifically, image data obtained by masking an areacorresponding to a position superimposed on the display unit 330 in theimage display unit 20 is output by the PC 300 to the connecting device10. Thus, display corresponding to the virtual image VD wider than thedisplay unit 330 of the PC 300 can be performed on the image displayunit 20 of the HMD 100. Thus, the display unit 330 of the PC 300 can beexpanded virtually by the HMD 100.

Additionally, the control unit 310 is configured to generate image dataobtained by removing an area displayed by the display unit 330 from thevirtual image VD, and cause the I/F unit 341 to output the image data.The control unit 310 is configured to cause the image display unit 20provided in the HMD 100 to display an image for display output bycontrol of the control unit 310, around the display unit 330 visuallyrecognized as the outside scene. That is, when the virtual image VDcorresponding to the virtual display area wider than the display unit330 of the PC 300 is displayed by the HMD 100, the virtual image VDobtained by removing an area displayed by the display unit 330 isdisplayed. Thus, display of the display unit 330 and display of the HMD100 can be combined not to overlap with each other. For example, theimage is displayed by the HMD 100 around the display unit 330 of the PC300 and thus, the display mode in which the display unit 330 isvirtually expanded can be achieved.

Additionally, the control unit 310 is configured to determine a relativeposition of the display unit 330 with respect to the image display unit20 of the HMD 100, and is configured to, based on the obtained relativeposition, perform the masking processing, generate image data, and causethe I/F unit 341 to output the image data. Accordingly, since the imagedata is generated in correspondence to the relative position of thedisplay unit 330 with respect to the image display unit 20, displaycorresponding to the position of the display unit 330 of the PC 300 canbe achieved easily by the HMD 100.

Additionally, the control unit 310 is configured to generate the virtualimage VD and determine a position of the virtual image VD based on aposition of the PC 300 in a real space. The control unit 310 isconfigured to adjust a display mode of the virtual image VD to bedisplayed by the image display unit 20 to correspond to a relativeposition between the virtual image VD and the image display unit 20.Accordingly, the virtual image VD can be displayed by the HMD 100 incorrespondence to the position of the display unit 330 of the PC 300 inthe real space. An effect such as supplementing or expanding the displayunit 330 can be obtained by this virtual image VD.

Additionally, the control unit 310 is configured to initialize a displaymode of the virtual image VD by using a position of the display unit 330detected as a reference when the control unit 310 detects that thedisplay unit 330 is present in the range visually recognized as anoutside scene in the image display unit 20. Accordingly, the displaymode can be adjusted in correspondence to whether the display unit 330of the PC 300 can be recognized visually as the outside scene in the HMD100. Additionally, even when a visual recognition state of the displayunit 330 of the PC 300 changes while the HMD 100 displays an image, anappropriate response to such a change can be made.

Additionally, the image display unit 20 includes the left display unit24 configured to emit imaging light toward the left eye LE of the user,and the right display unit 22 configured to emit imaging light towardthe right eye RE of the user in a state where the image display unit 20is worn on the head of the user. Then, the control unit 310 isconfigured to control a display position by the left display unit 24 anda display position by the right display unit 22 in correspondence to aposition of the display unit 330 visually recognized as an outside sceneof the image display unit 20, and adjust a convergence angle of an imageto be displayed by the image display unit 20. In this case, theconvergence angle of the image to be displayed by the HMD 100 isadjusted, and accordingly, a distance in which the display image of theHMD 100 is visually recognized can be made correspond to the position ofthe display unit 330 of the PC 300. Accordingly, the display by the HMD100 and the display by the display unit 330 of the PC 300 can becoordinated more appropriately.

Note that, in the processing in FIG. 11, the display unit 330 mayexcessively approach to the image display unit 20 to an extent that thedisplay unit 330 blocks a field of view passing through the imagedisplay unit 20. An example of such a case includes a case where theposition of the display unit 330 detected at step S23 is closer to theimage display unit 20 than the preset range. In this case, the controlunit 310 may stop image output to the connecting device 10 and return tostep S21.

In the above-described exemplary embodiment, the control unit 310 isconfigured to adjust image display positions in an image for the righteye RE and in an image for the left eye LE, and control a convergenceangle to control a visual recognition distance that the user perceivesto a display image of the image display unit 20.

When the image display unit 20 can adjust the visual recognitiondistance that the user perceives to the display image of the imagedisplay unit 20, by optical systems provided in the right display unit22 and the left display unit 24, this adjustment by control of theoptical systems may be performed instead of the control of theconvergence angle. That is, the optical system may be controlled suchthat the user perceives the display image of the image display unit 20being a portion of the virtual image VD as an image located at anidentical position to a position of the display unit 330. In this case,a distance in which the display image of the image display unit 20 isvisually recognized can be made correspond to the position of thedisplay unit 330 of the PC 300, and the display by the HMD 100 and thedisplay by the PC 300 can be coordinated more appropriately.

An example of the above-described optical system includes aconfiguration in which a lens movable in a left-right directionconnecting the right eye RE and the left eye LE is provided. In thisconfiguration, the lens movable in the left-right direction is disposedin each of the right display unit 22 and the left display unit 24. Thelenses may be disposed, for example, between the right optical system251 and the half mirror 261, and between the left optical system 252 andthe half mirror 281, respectively. Additionally, the lens may bedisposed inside each of the right optical system 251 and the leftoptical system 252. Since a position of imaging light entering each ofthe right eye RE and the left eye LE moves in the left-right directionaccording to the position of the lens, the convergence angle can bechanged. In this example, since the control unit 120 is configured tocontrol the movement of the lens, and the control unit 310 is configuredto output image data, and control data for specifying the lens positionto the connecting device 10, a convergence angle calculated by thecontrol unit 310 can be achieved.

Additionally, as an optical system used by the image display unit 20 tooutput imaging light, a holography display device may be provided. Inthis case, the image display unit 20 includes, for example, a lightsource, and a Spatial Light Modulator (SLM) configured to modulate lightfrom the light source. The SLM can use, for example, a reflection typespatial light phase modulator using liquid crystal (Liquid Crystal OnSilicon (LCOS)-SLM). In this configuration, the connecting device 10outputs display data to the SLM, and accordingly, reference lightemitted from the light source is modulated by the SLM to form astereoscopic image. In this case, image data output by the PC 300 to theconnecting device 10, or display data generated by the displaycontroller 122 from the image data output by the PC 300 are adjusted andthus, a focal distance of the stereoscopic image formed by the imagedisplay unit 20 can be adjusted. Thus, instead of the control foradjusting the convergence angle in the exemplary embodiment, the focaldistance of the stereoscopic image is controlled in the image displayunit 20 configured as the holography display device and thus, a distanceto which the user visually recognizes an image displayed by the imagedisplay unit 20 can be adjusted.

Additionally, a configuration in which a dimming shade is provided on anouter side of the image display unit 20 may be adopted. The dimmingshade is a plate shaped electronic device including a liquid crystalpanel or the like, and having light transmittance changing according tocontrol of the control unit 310. When the dimming shade is provided onthe image display unit 20, outside light OL passing through the dimmingshade and further passing through the right light-guiding plate 26 andthe left light-guiding plate 28 enters the right eye RE and the left eyeLE of the user wearing the image display unit 20. The user visuallyrecognizes the outside scene by the outside light passing through thedimming shade and the image display unit 20. Therefore, according totransmittance (light transmittance) of the dimming shade, visibility atwhich the user visually recognizes the outside scene changes.

When the control unit 310 executes the masking process, to enhancevisibility of an area subjected to the masking process, the control unit310 may perform control for enhancing the transmittance of the area inthe dimming shade.

In the above-described first exemplary embodiment, the configuration inwhich a position of the display unit 330 in a display range of the imagedisplay unit 20 is specified, and an image is displayed by the imagedisplay unit 20 in correspondence to the position of the display unit330 is described. That is, the image is displayed in correspondence tothe relative position between the image display unit 20 and the displayunit 330. The invention is not limited to this configuration, and forexample, a position of the PC 300 in a display range of the imagedisplay unit 20 may be detected or specified by using a portion or allof appearance of the PC 300 as a reference.

Namely, the control unit 310 determines, at step S22, whether at least aportion of the appearance of the PC 300 is present in the captured imagedata of the camera. At step S22, the control unit 310 detects, forexample, an image of the PC 300 from the captured image data by patternmatching. Here, the control unit 310 may detect a shape of a corner of ahousing of the PC 300, a shape of a border portion between the displayunit 330 and the housing in the PC 300, a shape of a back surface sideof the PC 300, a painting color of a back surface of the PC 300, or thelike from the captured image of the camera 61 by the pattern matching.Additionally, the control unit 310 may detect a marker attached or fixedto the housing of the PC 300 from the captured image of the camera 61.As the marker, for example, an image code such as a bar code and a twodimensional code, a character, or other images can be used.Additionally, a pattern or a shape of an outer packaging of the PC 300may be a pattern or a shape functioning as a marker. For example, the PC300 is not limited to a plane shape illustrated in FIG. 1, and may be ashape having a protrusion or a concave, or may include a configurationin which the control unit 310 detects the protrusion or the concave. Inthis case, the control unit 310 may specify a position or an attitude ofthe PC 300 from the captured image data, and may determine a relativeposition of the PC 300 with respect to the image display unit 20.Additionally, the control unit 310 may perform processing fordetermining the relative position of the PC 300 with respect to theimage display unit 20, and based on the obtained relative position,determining a relative position between the image display unit 20 andthe display unit 330.

2. Second Exemplary Embodiment

FIG. 12 is a view illustrating an example of a display mode in a secondexemplary embodiment to which the invention is applied.

The second exemplary embodiment is an example in which a virtual imageVD different from the virtual image VD in the first exemplary embodimentis formed by a display system 1 including a common configuration to theconfiguration of the display system 1 in the first exemplary embodiment.

The virtual image VD illustrated in FIG. 12 is, in a manner similar tothe virtual image VD illustrated in FIG. 8, an image having a sizelarger than a size of an image displayed by a display unit 330. In thesecond exemplary embodiment, a control unit 310 is configured to disposeall the image displayed by the display unit 330 in a virtual displayarea to generate the virtual image VD.

As illustrated in FIG. 12, the image displayed on all the virtual imageVD includes identical contents to all the image displayed by the displayunit 330. The control unit 310 is configured to perform resolutionconversion to generate the virtual image VD when resolution of a virtualdisplay area does not coincide with resolution of the image displayed bythe display unit 330 or display resolution of the display unit 330.

The virtual image VD is divided into areas VD1, VD2, VD3, and VD4. Aportion obtained by dividing a display image of the display unit 330 isassigned to each of the areas VD1 to VD4. In the virtual image VD inFIG. 12, the display image of the display unit 330 is divided by four,and the divided areas are displayed in the areas VD1, VD2, VD3, and VD4,respectively.

In processing for cutting out (extracting) the range to be displayed byan image display unit 20 from the virtual image VD (step S29 and stepS36 in FIG. 11), the control unit 310 cuts out any of the areas VD1 toVD4 as a unit. That is, image data of any of the areas VD1 to VD4 isoutput to a connecting device 10.

Then, in processing for determining whether a visual field VR has moved(step S33), in a case where the visual field VR has crossed any ofborders of the areas VD1 to VD4, the control unit 310 determines thatthe visual field VR has moved. In this determination, in a case wherethe center of the visual field VR has moved across any of the borders ofthe areas VD1 to VD4, namely, in a case where the center has moved fromany of the areas VD1 to VD4 to the area adjacent, the control unit 310determines positively.

In the second exemplary embodiment, a PC 300 is configured to cause thedisplay unit 330 to display an image corresponding to the display unit330, generate the virtual image VD corresponding to the display image ofthe display unit 330, and cause image data of an image including atleast a portion of the virtual image VD to be output. The display system1 is configured to cause the image display unit 20 provided in an HMD100 to display an image based on image data output by control of thecontrol unit 310, in correspondence to a position of the display unit330 visually recognized as an outside scene. Accordingly, at least aportion of the image displayed by the display unit 330 of the PC 300 canbe displayed by the image display unit 20, in correspondence to theposition of the display unit 330 visually recognized as the outsidescene in the image display unit 20. Thus, a display image of the displayunit 330 and an image displayed by the HMD 100 can be coordinated.

Additionally, the control unit 310 is configured to cause the I/F unit341 to output an image obtained by cutting out a portion of the virtualimage VD, and thus a portion of the display image of the display unit330 can be expanded and displayed by the image display unit 20.

Additionally, the control unit 310 is configured to determine a relativeposition of the display unit 330 with respect to the image display unit20 of the HMD 100, and based on the relative position determined,generate image data. Accordingly, display corresponding to the positionof the display unit 330 of the PC 300 can be achieved easily by the HMD100 including the image display unit 20.

3. Third Exemplary Embodiment

Each of FIG. 13 and FIG. 14 is a flowchart illustrating an operation ofa display system 1 in a third exemplary embodiment to which theinvention is applied. FIG. 13 illustrates an operation of a PC 300, andFIG. 14 illustrates an operation of an HMD 100.

The third exemplary embodiment is an example in which an operationdifferent from the operation in the first exemplary embodiment isexecuted by the display system 1 including a common configuration to theconfiguration of the display system 1 in the first exemplary embodiment.

In the third exemplary embodiment, while processing for detecting arelative position of the PC 300 with respect to an image display unit 20is executed by the PC 300, processing for cutting out a portion of avirtual image VD generated by the PC 300 is executed by the HMD 100.

In the flowchart illustrated in FIG. 13, common processing to theprocessing in FIG. 11 is denoted by an identical step number anddescription of the common processing will be omitted.

A control unit 310 is configured to execute processing at steps S21 toS28. At step S28, the control unit 310 specifies a position of a displayunit 330 in the range in which the image display unit 20 displays animage (step S28). Here, the control unit 310 starts processing foroutputting position data indicating the position of the display unit 330specified at step S28, together with image data of the virtual image VDto the HMD 100 (step S51). The position data output by the control unit310 at step S51 is data for cutting the image present in the rangedisplayed by the image display unit 20 out of the virtual image VD.

The control unit 310 determines in a manner similar to step S32 (FIG.11) whether a visual field VR has moved (step S52). At step S52, animage adjustment unit 315 performs the determination based on adetection value of a position detection unit 313. Additionally, stepsS53 to S57 described below are executed by the image adjustment unit315.

When the control unit 310 determines that the visual field VR has moved(step S52; YES), the control unit 310 determines whether a position ofthe PC 300 is out of the range in which the image display unit 20displays an image (step S53). At step S53, the control unit 310 maydetermine whether the position of the display unit 330 is out of therange in which the image display unit 20 displays an image. At step S53,the control unit 310 performs the determination based on a relativeposition and a relative direction between the image display unit 20 andthe display unit 330.

When it is determined that the position of the PC 300 is out of thedisplay range of the image display unit 20 (step S53; YES), positiondata to be output to the HMD 100 is changed in correspondence to thevisual field VR having moved (step S54), and the processing returns tostep S52.

When the control unit 310 determines that the position of the PC 300 isnot out of the display range of the image display unit 20 (step S53;NO), the control unit 310 determines whether the PC 300 has moved froman outside of the display range of the image display unit 20 into therange of the image display unit 20 t (step S55).

In a case where, owing to the movement of the visual field VR, thedisplay unit 330 enters the display range from the state where thedisplay unit 330 has been out of the display range of the image displayunit 20 (step S55; YES), the control unit 310 determines that display ofthe HMD 100 needs to be initialized (step S56). The control unit 310returns to step S23, and executes again the processing for startingoutput of the image data. This initialization processing is similar tothe processing described at step S38, and in the operation exampleillustrated in each of FIG. 13 and FIG. 14, the image display unit 20performs the initialization.

When it is determined that the PC 300 has not moved from the outside ofthe display range of the image display unit 20 into the range of theimage display unit 20 (step S55; NO), the control unit 310 shifts tostep S54 and changes the position data.

At step S52, in a manner similar to step S33, in any of a case where theimage display unit 20 has moved, and a case where the PC 300 has moved,as long as a relative position between the image display unit 20 and thePC 300 is changed, it is determined that the visual field VR has moved.

When the control unit 310 determines that the visual field VR has notmoved (step S52; NO), the control unit 310 determines whether to end thedisplay (step S57). For example, when display end is instructed by anoperation on the display unit 330 or the like (step S57; YES), thecontrol unit 310 stops outputting the image data to a connecting device10 and ends this processing. On the other hand, when the display is tobe continued (step S57; NO), the control unit 310 returns to step S52.

In the operation illustrated in FIG. 14, the control unit 120 of the HMD100 starts processing for acquiring data of the virtual image VD outputby the PC 300 and the position data (step S61). The control unit 120specifies, based on the acquired position data, a position of thedisplay unit 330 in the range in which the image display unit 20displays an image (step S62).

The control unit 120 executes processing for cutting an image present inthe range to be displayed by the image display unit 20 out of thevirtual image VD (step S63). The control unit 120 applies maskprocessing to the cut out image (step S64). The range in which themasking processing is applied at step S64 is identical to the range atstep S30 (FIG. 11), and is the range in which the display unit 330 issuperimposed on the image to be displayed by the image display unit 20,or the range including a portion superimposed on the display unit 330.

The control unit 120 sets a convergence angle of the image subjected tothe masking processing (step S65). At step S65, in a manner similar tothe processing executed by the control unit 310 at step S31 (FIG. 11),the control unit 120 sets the convergence angle of the image such that aportion of the virtual image VD to be displayed by the image displayunit 20 is visually recognized as a plane including the display unit330. The control unit 120 generates image data including image data ofan image to be displayed by a right display unit 22 of the image displayunit 20, and image data of an image to be displayed by a left displayunit 24, and starts display (step S66). Here, the control unit 120adjusts a position of the virtual image VD in an image for the rightdisplay unit 22, and a position of the virtual image VD in an image forthe left display unit 24 and thus, the control unit 120 can adjust aconvergence angle of a left eye LE and a right eye RE of a user.

The control unit 120 determines whether the position data output by thePC 300 has been changed (step S67). When the control unit 120 determinesthat the position data has been changed (step S67; YES), the controlunit 120 determines whether a position of the PC 300 is out of the rangein which the image display unit 20 displays an image (step S68). Theprocessing at step S68 is, for example, similar to the processingexecuted by the control unit 310 at step S34 (FIG. 11).

When the control unit 120 determines that the position of the PC 300 isout of the display range of the image display unit 20 (step S68; YES),the control unit 120 cancels the masking processing applied at step S64(step S69). The control unit 120 changes, in correspondence to theposition data changed, the range to be cut out of the virtual image VD(step S70), and returns to step S67.

When the control unit 120 determines that the position of the PC 300 isnot out of the display range of the image display unit 20 (step S68;NO), the control unit 120 determines whether the PC 300 has moved froman outside of the display range of the image display unit 20 into therange of the image display unit 20 (step S71). In a case where the PC300 enters the display range from the state where the PC 300 has beenout of the display range of the image display unit 20 (step S71; YES),the control unit 120 initializes display of the image display unit 20(step S72). The initialization at step S72 is, for example, similar tothe initialization at step S38 (FIG. 11), and the control unit 120resets application/canceling of the masking process, the range to be cutout of the virtual image VD, or the like to an initial state. Thecontrol unit 120 returns to step S61, and performs again the processingfor starting the display.

In a case where the PC 300 has not moved from the outside of the displayrange of the image display unit 20 into the range of the image displayunit 20 (step S71; NO), the control unit 120 shifts to step S70.

Additionally, when the control unit 120 determines that the positiondata has not been changed (step S67; NO), the control unit 120determines whether to end the display (step S73). For example, whendisplay end is instructed by a command input from the PC 300 or anoperation on the operating unit 140 or the like (step S73; YES), thecontrol unit 120 stops the display by the image display unit 20 and endsthis processing. On the other hand, when the display is to be continued(step S73; NO), the control unit 120 returns to step S67.

According to this third exemplary embodiment, the PC 300 is configuredto specify the relative position of the PC 300 with respect to the imagedisplay unit 20, and perform processing for determining the position ofthe PC 300 in the display range of the image display unit 20, and theHMD 100 is configured to cut an image out of the virtual image VD tocause the image display unit 20 to display the cut out image.Accordingly, since the PC 300 and the HMD 100 are configured to mutuallyshare and execute the processing, the configuration of the HMD 100 issimplified, and thus the HMD 100 can be reduced in a weight and a size.Additionally, since the image display unit 20 is configured to executethe processing for cutting the image out of the virtual image VD, forexample, processing such as setting the convergence angle can beexecuted rapidly in the image display unit 20.

In the operation described in the third exemplary embodiment, processingsimilar to steps S33 to S38 (FIG. 11) may be executed. In this case, theprocessing at steps S33, S34, and S37 is preferably executed by thecontrol unit 310. Additionally, preferably, the control unit 120acquires processing results of steps S33, S34, and S37 executed by thecontrol unit 310, and executes the processing at steps S35, S36, andS38.

4. Fourth Exemplary Embodiment

FIG. 15 is a block diagram of respective components constituting adisplay system 1A according to a fourth exemplary embodiment.

The display system 1A includes a configuration in which the HMD 100 inthe display system 1 described in the first exemplary embodimentincludes a control device 10A instead of the connecting device 10. Thecontrol device 10A includes a control unit 121, and the control unit 121is configured to process image data input from a PC 300. Except for theconfigurations of the control unit 121 and the related components, thedisplay system 1A includes components similar to the components of thedisplay system 1. The components common to the components of the displaysystem 1 are denoted by identical reference signs and description of thecommon components will be omitted.

In a manner similar to the control unit 120 (FIG. 5), the control unit121 includes a processor (not illustrated) such as a CPU or amicrocomputer, and is configured to cause the processor to execute aprogram to control each component of the HMD 100 in cooperation ofsoftware and hardware. Additionally, the control unit 121 may includeprogrammed hardware.

The control unit 121 is connected with a non-volatile storage 130, anoperating unit 140, and a connecting unit 145. Additionally, the controlunit 121 includes a position detection unit 123, and an image adjustmentunit 125. In a manner similar to the position detection unit 313, theposition detection unit 123 is configured to detect a position of theHMD 100. More specifically, the position detection unit 123 isconfigured to detect a relative position between the PC 300 and an imagedisplay unit 20, based on an output value of a sensor provided in theimage display unit 20. Additionally, for example, the processing inwhich the control unit 310 determines the distance from the capturedimage data of the camera 61 to the object to be captured as described inthe first exemplary embodiment can also be executed by the control unit121.

The relative position detected by the position detection unit 123 may bea position in a space in which the image display unit 20 and the PC 300exist. Additionally, the relative position detected by the positiondetection unit 123 may include a relative direction between the imagedisplay unit 20 and a display unit 330. For example, the relativeposition detected by the position detection unit 123 may be informationindicating the position and/or the direction of the display unit 330with respect to the image display unit 20. Additionally, the relativeposition detected by the position detection unit 123 may be, forexample, information indicating the position and/or the direction of theimage display unit 20 with respect to the PC 300. Additionally, therelative position detected by the position detection unit 123 mayinclude, for example, coordinates in a three dimensional coordinatesystem set in a space in which the image display unit 20 and the PC 300exist.

The image adjustment unit 125 is configured to perform image processingfor image data input to an I/F unit 110. The image processing executedby the image adjustment unit 125 is similar to the processing of theimage adjustment unit 315 described in the first exemplary embodimentand the second exemplary embodiment.

For example, the image adjustment unit 125 is configured to performprocessing such as resolution conversion (scaling), frame rateconversion, tone correction, and data format change on the image datainput to the I/F unit 110, in correspondence to specifications of theHMD 100.

In a manner similar to the display system 1, the display system 1A isconfigured to execute the calibration processing illustrated in FIG. 7,and the operation illustrated in FIG. 11.

When the display system 1A executes the operation in FIG. 7, steps S13to S15 are executed by the position detection unit 123. Additionally,when the display system 1A executes the operation in FIG. 11, theposition detection unit 123 executes steps S21 and S23, and the imageadjustment unit 125 executes steps S22, S24 to S39.

In the fourth exemplary embodiment, the image adjustment unit 125 isconfigured to set a virtual display area corresponding to the virtualimage VD, and dispose image data output by the PC 300 in the virtualdisplay area to generate the virtual image VD. The image adjustment unit125 is configured to cut the range to be displayed by the image displayunit 20 out of the virtual image VD, apply the masking processing asnecessary, and cause the image display unit 20 to display the image.

Namely, an HMD 100A includes the control unit 121 configured todetermine a relative position of the display unit 330 with respect tothe image display unit 20, process image data based on the relativeposition determined, and cause the image display unit 20 to display animage corresponding to a position of the display unit 330 of an outsidescene.

Accordingly, since the HMD 100A determines the relative position of thedisplay unit 330 with respect to the image display unit 20, andgenerates an image for display to correspond to this relative position,display corresponding to the position of the display unit 330 can beachieved without increasing a load on the PC 300.

Additionally, in a configuration in which the PC 300 mirrors and outputsvia the HDMI cable 2A an image identical to the image displayed by thedisplay unit 330, the HMD 100A adjusts a display image of the imagedisplay unit 20. Namely, the HMD 100A changes, in correspondence to adirection of the image display unit 20 and a relative position betweenthe image display unit 20 and the PC 300, a display mode of the image tobe displayed by the image display unit 20.

Therefore, the display system 1A can achieve the effect of the displaysystem 1 described in the first exemplary embodiment. Further, since theimage data can be processed in the HMD 100A, when general-purposeequipment configured to output image data is connected instead of the PC300, there is an advantage that a display mode can be changed incorrespondence to the position or the direction of the image displayunit 20.

Note that the invention is not limited to the configurations in theabove-described exemplary embodiments, and the invention can beimplemented in various aspects without departing from the gist of theinvention.

For example, in the above-described exemplary embodiments, theconfiguration in which the user visually recognizes the outside scenethrough the display unit is not limited to the configuration in whichthe right light-guiding plate 261 and the left light-guiding plate 281transmit the outside light. For example, the invention is applicable toa display device configured to display an image in a state where anoutside scene cannot be recognized visually. Specifically, the inventionis applicable to a display device configured to display a captured imageof the camera 61 for capturing an outside scene, an image or CGgenerated based on this captured image, an image based on image datastored in advance or based on image data input from an outside, or thelike. This kind of display device can include a so-called closed typedisplay device in which an outside scene cannot be recognized visually.Additionally, as described in the above-described exemplary embodiments,AR display in which an image is superimposed on a real space and isdisplayed in the real space, or Mixed Reality (MR) display in which acaptured image in a real space and a virtual image are combined may beused. Alternatively, the invention is applicable to a display deviceconfigured to perform no processing of Virtual Reality (VR) display fordisplaying a virtual image. For example, a display device configured todisplay image data input from an outside or an analogue image signal isalso, as a matter of course, encompassed as the application of theinvention.

Additionally, instead of the image display unit 20, for example, animage display unit of another type such as an image display unit worn asa hat may be adopted, as long as the image display unit includes adisplay unit configured to display an image in correspondence to a lefteye of a user, and a display unit configured to display an image incorrespondence to a right eye of the user. Additionally, the displaydevice in the invention may be configured, for example, as a headmounted display mounted on a vehicle such as a car, and an airplane.Additionally, the display device in the invention may be configured, forexample, as a head mounted display built in a body protector such as ahelmet. In this case, a portion positioning a position with respect to abody of a user, and a portion positioned with respect to such a portioncan be mounted parts.

Further, the HMD 100A in the fourth exemplary embodiment may include aconfiguration in which the control device 10A and the image display unit20 are integrally constituted and are worn on a head of a user.Additionally, a note type computer, a tablet computer, a portableelectronic device including a game machine, a portable telephone, asmartphone, and a portable media player, other dedicated equipment, orthe like may be used as the control device 10A. Additionally, in theabove-described exemplary embodiments, a configuration in which theconnecting device 10 and the image display unit 20 are connected via aradio communication line may be adopted.

Additionally, equipment connected with the HMD 100 or 100A in thedisplay system 1 is not limited to the PC 300. The equipment may be, forexample, a stationary television set, or a monitor for a stationarypersonal computer. Additionally, a projector configured to project animage on a display surface may be used instead of the PC 300, and inthis case, the projection surface on which the projector projects theimage corresponds to the first display unit. Additionally, other thanthe above-described equipment, instead of the PC 300, a portable or astationary electronic device can be used. The PC 300 or variouselectronic devices used instead of the PC 300 may be connectedwirelessly with the HMD 100 or 100A. For example, instead of theconnector 11A, a radio image communication interface such as Miracast(trade name) or WirelessHD (trade name) may be used. Additionally,instead of the connector 11B, a wireless LAN (including WiFi (tradename)) may be used, or Bluetooth (trade name) may be used.

Additionally, at least a portion of each function block illustrated inthe block diagram may be configured to be achieved with hardware, or maybe configured to be achieved in cooperation of hardware and software,and the invention is not limited to the configuration in whichindependent hardware resources are disposed as illustrated in thefigure.

The entire disclosure of Japanese Patent Application No.:2017-246853,filed Dec. 22, 2017 and 2018-172773, filed Sep. 14, 2018 are expresslyincorporated by reference herein.

1. A display system comprising: an electronic device including: aprocessor configured to generate a first image and a second image thatis different from the first image, and a first image display configuredto display the first image; and a head mounted type display deviceconfigured to communicate with the processor of the electronic device,the head mounted display including: a camera configured to capture anoutside scene, and a second image display configured to display thesecond image overlapping the outside scene, wherein: the processor ofthe electronic device is configured to: acquire a position of theelectronic device in the outside scene captured by the camera; and applya mask to a portion overlapping the electronic device when the secondimage includes a portion overlapping the electronic device, the maskincluding changing color data of pixels to a dark color or a blackcolor.
 2. The display system according to claim 1, wherein the secondimage is wider than the first image, and the second image includes apart of the first image.
 3. The display system according to claim 1,wherein the processor is configured to: cause the second image displayto display the part of the first image in the second image to overlapthe electronic device.
 4. The display system according to claim 1,wherein the processor is configured to: cause the second image displayto display an area around the position of the electronic device.
 5. Thedisplay system according to claim 3, wherein the processor is configuredto: determine a relative position of the electronic device withrespective to the second image; and change a position of the secondimage being displayed on the second image display to maintain therelative position when changing the position of the electronic device.6. The display system according to claim 5, wherein the processor isconfigured to: generate a third image that is different from the firstimage and the second image; and cause the second image display todisplay the third image when changing the position of the electronicdevice.
 7. A display system method for use with an electronic device anda head mounted display, the display system method comprising: generatinga first image and a second image that is different from the first image;displaying the first image in a first image display of the electronicdevice; capturing an outside scene by a camera of the head mounteddisplay; displaying the second image overlapping the outside scene in asecond image display of the head mounted display; acquiring a positionof the electronic device in the outside scene; and when the second imageincludes a portion overlapping the electronic device, applying a mask tothe portion overlapping the electronic device, the mask includingchanging color data of pixels to a dark color or a black color.
 8. Adisplay system comprising: an electronic device comprising: a processorconfigured to generate a first image and a second image that isdifferent from the first image; and a first image display configured todisplay the first image, a head mounted display that communicates withthe processor of the electronic device, the head mounted displayincluding: a camera configured to capture an outside scene; and a secondimage display configured to display the second image overlapping theoutside scene, wherein: the processor of the electronic device isconfigured to: acquire a position of the electronic device in theoutside scene captured by the camera, determine a relative position ofthe second image with respect to the head mounted display, cause thesecond image display to display the second image based on the positionof the electronic device, and change a position of the second imagebeing displayed on the second image display to maintain the relativeposition when the position of the electronic device is changed.